Image processing system, image processing method, and program

Abstract

To provide a technique for supporting a selection of an advertisement to be arranged in a virtual viewpoint image.SOLUTION: Information of a virtual viewpoint set in a virtual space is acquired. Area information indicating an advertisement area arranged in the virtual space is acquired. Based on the information of a virtual viewpoint and the area information, an advertisement effect of the advertisement area is estimated. Information indicating the advertisement effect estimated for the advertisement area is presented. After the information indicating the advertisement effect estimated for the advertisement area is presented, a designation of an advertisement content to be arranged in the advertisement area is acquired. A virtual viewpoint image according to the information of the virtual viewpoint is generated for the virtual space in which the advertisement content designated in the advertisement region is arranged.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an image processing method, and a program, and more particularly to an image processing device that generates an image from an arbitrary virtual viewpoint. [Background technology]

[0002] In recent years, technology that generates virtual viewpoint images from any viewpoint, rather than just images from a camera installation position, has been attracting attention. Such virtual viewpoint images can be generated using multiple viewpoint images obtained by installing multiple cameras in different positions and synchronously capturing images from multiple viewpoints. This technology allows a scene to be viewed from various positions or angles. For example, highlight scenes of a soccer or basketball game can be viewed from the field during the game. Such images, which were previously impossible to achieve, allow users to experience a highly realistic sensation.

[0003] The virtual viewpoint image can be generated by processes such as foreground / background separation, 3D shape estimation, 3D model generation, and rendering. Such processes can be performed on a server to which multiple viewpoint images from multiple cameras are transferred. The server can then transfer the generated virtual viewpoint image to a user terminal for viewing.

[0004] It is also possible to insert advertisements into such virtual viewpoint images. For example, it is possible to insert virtual billboards that do not actually exist into virtual viewpoint images. Such virtual advertisements are not subject to physical constraints. Therefore, compared to regular advertisements, they have a higher degree of freedom in placement. On the other hand, virtual advertisements do not exist in real stadiums, etc., and advertising effectiveness cannot be achieved unless they are displayed in the virtual viewpoint image. In view of these issues, Patent Document 1 proposes a technology that determines the amount of money charged for an advertisement based on the size of the advertisement displayed in the virtual viewpoint image when a virtual viewpoint image from a viewpoint in accordance with a user's instructions is generated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-101897 Summary of the Invention [Problem to be solved by the invention]

[0006] According to the technology described in Patent Document 1, the amount charged is determined based on factors such as the size of an advertisement placed in a virtual space on a virtual viewpoint image. Meanwhile, advertisers must select the advertisement to place in the virtual space. Furthermore, when there are multiple options for advertisement placement, advertisers must select the placement location of the advertisement. It has not been easy for advertisers to select an advertisement or its placement location in a way that enhances the effectiveness of their advertisement.

[0007] The present invention provides a technique for supporting the selection of advertisements to be placed in a virtual viewpoint image. [Means for solving the problem]

[0008] An image processing system according to an embodiment includes the following configuration: a viewpoint acquisition means for acquiring information about a virtual viewpoint set in a virtual space; an area acquisition means for acquiring area information indicating an advertising area arranged in the virtual space; an estimation means for estimating an advertising effectiveness for the advertising area based on the virtual viewpoint information and the area information; a presentation means for presenting information indicating the estimated advertising effectiveness for the advertising area; an advertisement acquisition means for acquiring, after presenting information indicating the estimated advertising effectiveness for the advertising area, a designation of advertising content to be placed in the advertising area; a generating means for generating a virtual viewpoint image according to the information on the virtual viewpoint for the virtual space in which the advertising content designated in the advertising area is placed; Equipped with. [Effects of the Invention]

[0009] It can assist in the selection of advertisements to be placed in the virtual viewpoint image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of an image processing system according to an embodiment. [Figure 3] 1 is a flowchart of an image processing method according to an embodiment. [Figure 4] FIG. 10 is a schematic diagram showing an example of a virtual signboard. [Figure 5] 3A and 3B are schematic diagrams showing examples of virtual viewpoint images and virtual camera parameters; [Figure 6] Schematic diagram showing a UI for modifying a virtual advertisement. [Figure 7] Schematic diagram showing how virtual advertisements are synthesized. [Figure 8] 10 is a schematic diagram showing a UI for transferring rights to place a virtual advertisement. [Figure 9] 1 is a flowchart of a method for transferring rights according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] An image processing system according to an embodiment will be described with reference to the drawings. FIG. 1 is a block diagram schematically illustrating the hardware configuration of an image processing system 100 according to an embodiment. The image processing system 100 generates a virtual viewpoint image that represents the view from a specified virtual viewpoint. The virtual viewpoint image may be an image of a virtual space in which a model is placed, as viewed from the virtual viewpoint. The image processing system 100 can generate the virtual viewpoint image based on multiple captured images captured by multiple imaging devices. In other words, a model to be placed in a virtual space can be generated using multiple captured images from multiple viewpoints.

[0013] In this specification, a virtual viewpoint image is also referred to as a free viewpoint video. A virtual viewpoint image is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user. For example, a virtual viewpoint image may be an image corresponding to a viewpoint selected by a user from multiple candidates. The following mainly describes a case where a virtual viewpoint is specified by a user operation. However, the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. In this specification, a virtual viewpoint image is a moving image or a still image. The following mainly describes a case where the virtual viewpoint image is a moving image. In other words, at least one of the virtual viewpoint and the model placed in the virtual space may change over time. Each frame image of a moving virtual viewpoint image may be an image of the virtual space in which the model is placed, as seen from the virtual viewpoint at the respective time. In this specification, a moving virtual viewpoint image is particularly referred to as a virtual viewpoint video.

[0014] The image processing system 100 includes a CPU 101, a main storage device 102, an auxiliary storage device 103, an external I / F 104, and a GPU 105. These components are connected via an internal bus 106 so that they can communicate with each other.

[0015] The CPU 101 is a central processing unit that controls the entire image processing system 100. The CPU 101 can perform various calculations and decisions, image processing, and data input / output.

[0016] The main memory device 102 is a storage device that functions as a work area or a temporary data storage area for the CPU 101. The main memory device 102 can be implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0017] The auxiliary storage device 103 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, 3D shape data, or 2D maps. The auxiliary storage device 103 can be implemented using a storage medium such as a non-volatile memory such as a read-only memory (ROM) or flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or tape media. To increase capacity or speed, a combination of multiple storage media may be used as the auxiliary storage device 103. These storage media may be logically combined into one, for example, as in a RAID.

[0018] The external I / F 104 is an interface with an external device. The external I / F 104 may be a communication interface used for communication with an external device such as a camera, a PC, or a server. The external I / F 104 may also be a general-purpose input / output interface. For example, the external I / F 104 can be connected to external devices such as a joystick, a keyboard, or a mouse for accepting various operations by a user. The external I / F 104 can also be connected to external storage for file input / output. The external I / F 104 may have a connection terminal for a physical cable such as InfiniBand, Ethernet, or Universal Serial Bus (USB). The external I / F 104 may also be an interface for wireless connection such as a wireless LAN or Bluetooth.

[0019] The GPU 105 is a computing device for performing image processing calculations at high speed. The GPU 105 may have a function of outputting a video signal to an external video display device such as a television. The GPU 105 can also perform a process of rendering a virtual viewpoint image. The GPU 105 may also have a function of generating shape data or color information used to generate the virtual viewpoint image. The GPU 105 can output the video signal via a video terminal such as DVI, HDMI (registered trademark), Display Port, or SDI. The GPU 105 may include a main memory device separate from the main memory device 102. The GPU 105 may also perform some or all of the calculations performed by the CPU 101. The GPU 105 may not have a function of outputting a video signal. In this case, the external I / F 104 may further have a video output function or a data output function required for video output. In this case, the GPU 105 may only perform image processing and calculations such as shape data or color information used to generate the virtual viewpoint image.

[0020] The internal bus 106 may be a general-purpose bus such as PCI Express. However, the type of the internal bus 106 is not limited. For example, a USB may be used as the internal bus 106. Alternatively, a non-standardized, dedicated communication bus that exhibits the required performance may be used as the internal bus 106. Each component can perform bidirectional communication via the internal bus 106.

[0021] As described above, the image processing system 100 according to one embodiment can be realized by a computer including one or more processors and one or more memories. In one embodiment, a processor such as the CPU 101 or the GPU 105 executes a program stored in a memory such as the main storage device 102 or the auxiliary storage device 103. In this way, the functions of the units shown in FIG. 2 and the like, which will be described later, and image processing according to the flowchart in FIG. 5 can be realized. Alternatively, some or all of the functions of the image processing system 100 may be realized by dedicated hardware.

[0022] It should be noted that the configuration shown in FIG. 1 is merely one example of the configuration of the image processing system 100. For example, at least some of the functions shown in FIG. 1 may be realized by other hardware. As an example, the auxiliary storage device 103 may be connected to the outside of the image processing system 100 via an external I / F 104. Also, the main storage device 102 and the auxiliary storage device 103 may be realized by a single piece of hardware. Furthermore, an FPGA, a DSP, or an image processing LSI may be used instead of the GPU 105. In this way, various hardware may be used within the scope of the present invention.

[0023] Furthermore, the image processing system 100 does not need to be realized using a single device. For example, the image processing system 100 may be configured by a plurality of information processing devices. The plurality of information processing devices may have the same configuration. Furthermore, the plurality of information processing devices may have different configurations. The image processing system 100 according to one embodiment may be configured by a plurality of information processing devices connected via a network, for example.

[0024] 2 is a schematic diagram showing functional blocks of an image processing system 100 according to an embodiment. The image processing system 100 includes an image acquisition unit 201, an area information acquisition unit 202, a viewpoint acquisition unit 203, an advertising effect calculation unit 204, a user management unit 205, an advertising data acquisition unit 206, an advertising image generation unit 207, an advertising synthesis unit 208, an image output unit 209, and a display control unit 210.

[0025] The image acquisition unit 201 acquires a virtual viewpoint image. The image acquisition unit 201 may acquire the virtual viewpoint image from an external device. This external device may be, for example, a virtual viewpoint image generation server that generates the virtual viewpoint image. Hereinafter, the virtual viewpoint image acquired by the image acquisition unit 201 is referred to as an input virtual viewpoint image.

[0026] In this embodiment, the virtual viewpoint image is an image of a virtual space in which a model is placed, as viewed from a virtual viewpoint. The type of model is not limited. In this embodiment, the model includes a three-dimensional model of a subject in real space. As described above, such a three-dimensional model can be generated using multiple captured images of a subject from multiple viewpoints. The model may also include a model of a virtual object. Furthermore, the model may also include a background model.

[0027] The virtual viewpoint used to generate a virtual viewpoint image may be changeable. For example, the creator of the virtual viewpoint image can specify a virtual viewpoint. In this case, a virtual viewpoint image according to the specified virtual viewpoint is generated. The virtual viewpoint can be represented by parameters of a virtual camera. A virtual camera is a virtual camera that is different from an imaging device that captures an image of real space. The virtual camera is a concept for explaining a virtual viewpoint corresponding to a virtual viewpoint image. The position and orientation of the virtual viewpoint from which the virtual space is viewed can be represented as the position and orientation of the virtual camera. In other words, the virtual viewpoint image is an image simulating an image captured by a camera when it is assumed that the camera exists at the position of the virtual viewpoint set in the virtual space. Furthermore, viewpoint information representing the virtual viewpoint can be represented by external parameters and internal parameters of the virtual camera.

[0028] In this embodiment, the virtual viewpoint changes over time. In this specification, the camera parameters of the virtual viewpoint at each time are referred to as a virtual camera path. However, it is not necessary to use the concept of a virtual camera to realize the configuration of this embodiment. In one embodiment, for the virtual viewpoint, at least information representing a specific position in space and viewpoint information representing a specific orientation are set.

[0029] In this embodiment, the image acquisition unit 201 further acquires depth information of the input virtual viewpoint image. The depth information represents the distance from the virtual camera for each pixel of the virtual viewpoint image. The virtual viewpoint image may have a depth channel representing such depth information. For example, data of the virtual viewpoint image may be data having an R value, a G value, a B value, and a depth value for each pixel. On the other hand, the depth information may be metadata of the virtual viewpoint image.

[0030] Note that a virtual advertisement may or may not already be placed in the virtual viewpoint image acquired by the image acquisition unit 201. Even if a virtual advertisement is placed in the virtual viewpoint image, the virtual advertisement placed in the virtual viewpoint image can be changed by the process described below.

[0031] The area information acquisition unit 202 acquires area information indicating an advertising area arranged in a virtual space. In this embodiment, advertising content is displayed in the advertising area specified by the area information. The area information can indicate the position and size of the advertising area in the virtual space. The advertising area may be a two-dimensional area or a three-dimensional area. For example, the area information may be three-dimensional shape information indicating the position and shape in the virtual space. The area information acquisition unit 202 can acquire signboard information, which is information about a virtual signboard for displaying advertising content. The signboard information can include the above-mentioned area information. In this case, the area information indicates the area of ​​the virtual signboard. The signboard information may also include image placement information indicating the position where an image is to be pasted on the virtual signboard.

[0032] In this embodiment, the area information is set in advance in association with a virtual viewpoint image that has been generated in advance. For example, when generating a virtual viewpoint image, the creator of the virtual viewpoint image may set an advertising area in the virtual viewpoint image. Alternatively, the system may automatically set an advertising area in the virtual viewpoint image. Such a virtual viewpoint image may be generated by the image processing system 100 or another system.

[0033] The viewpoint acquisition unit 203 acquires information about a virtual viewpoint set in a virtual space (hereinafter, sometimes referred to as viewpoint information). In this embodiment, the information about the virtual viewpoint represents parameters of the virtual viewpoint for a virtual viewpoint image that has been generated in advance. For example, the viewpoint acquisition unit 203 can acquire viewpoint information that represents the virtual viewpoint used to generate an input virtual viewpoint image. The viewpoint information may indicate the position and orientation of the virtual viewpoint. The viewpoint information may also indicate the focal length or angle of view of the virtual viewpoint. For example, the viewpoint acquisition unit 203 can acquire camera parameters (e.g., external parameters and internal parameters) of a virtual camera corresponding to the input virtual viewpoint image.

[0034] The advertising effect calculation unit 204 estimates the advertising effect for the advertising area. The advertising effect calculation unit 204 can calculate the advertising effect of the virtual advertisement in the virtual viewpoint image generated by the advertisement synthesis unit 208. The advertising effect calculation unit 204 can estimate the advertising effect based on the viewpoint information acquired by the viewpoint acquisition unit 203 and the area information acquired by the area information acquisition unit 202. Note that in this specification, the advertising effect calculated in the embodiments described below does not depend on the content of the virtual advertisement. For this reason, hereinafter, the advertising effect for the advertising area may be referred to as the advertising effect of the virtual advertisement placed in the advertising area (or simply the advertising effect of the virtual advertisement).

[0035] For example, the advertising effect calculation unit 204 can estimate the advertising effect of an advertising area based on at least one of the size and tilt of the advertising area displayed in the virtual viewpoint image. The advertising effect calculation unit 204 can also estimate the advertising effect of an advertising area based on at least one of the display time of the advertising area in the virtual viewpoint video and the movement speed of the advertising area. The advertising effect calculation unit 204 can also estimate the advertising effect based on at least one of the resolution of the virtual viewpoint image, the compression rate of the virtual viewpoint image, and the frame rate of the virtual viewpoint video. The advertising effect calculation unit 204 can also estimate the advertising effect according to the movement speed of the advertising area in the virtual viewpoint video. These advertising effects can be statically determined based on viewpoint information, area information, and parameters of the virtual viewpoint image (e.g., resolution). In addition to these parameters, the advertising effect calculation unit 204 may also estimate the advertising effect based on dynamically changing parameters such as statistical information. The statistical information may be statistical information related to the playback of the virtual viewpoint image. For example, the statistical information may represent the number of playbacks of the virtual viewpoint image. The number of times the virtual viewpoint image is played may be the number of times the virtual viewpoint image is played per unit time, for example, within one minute immediately preceding the current time. The advertising effect calculation unit 204 can acquire or store such statistical information. A specific example of a method for calculating the advertising effect will be described later.

[0036] The display control unit 210 presents information indicating the advertising effectiveness estimated for the advertising area. The display control unit 210 can present the information indicating the advertising effectiveness via a display device connected via the external I / F 104 and a network, or a display device connected to the GPU 105. In this embodiment, the display control unit 210 generates a user interface (UI) used by a user of the image processing system 100. The display control unit 210 can display the generated UI on a display device connected to the GPU 105. The display control unit 210 can also present information indicating the advertising effectiveness on the UI. The user of the image processing system 100 can perform user input for advertising in the virtual viewpoint image via the UI generated by the display control unit 210.

[0037] In the following embodiments, a user operates the image processing system 100 via a web screen displayed on a browser. Such a web screen can be displayed by starting browser software on a PC or the like and entering a URL. This web screen can include a UI generated by the display control unit 210. However, the method of operating the image processing system 100 is not limited to this method. For example, the image processing system 100 can be operated using a dedicated application. Such an application can have a UI as described below.

[0038] The user management unit 205 performs user authentication. The user management unit 205 can manage users who advertise in virtual viewpoint images using the image processing system 100. In particular, the user management unit 205 can authenticate that a user has the authority to place advertising content in an advertising area. In this way, the user management unit 205 can manage users who have the right to change advertising images.

[0039] The advertisement data acquisition unit 206 acquires designation of advertisement content to be placed in the advertisement area. After the display control unit 210 presents information indicating the estimated advertisement effectiveness for the advertisement area, the advertisement data acquisition unit 206 can acquire designation of advertisement content to be placed in the advertisement area. With this configuration, a user who advertises can select advertisement content to be placed in the advertisement area by referring to the estimated advertisement effectiveness for the advertisement area. If the user management unit 205 has authenticated that the user has the right to change the advertisement image, the advertisement data acquisition unit 206 may accept designation of advertisement content to be placed in the advertisement area from the user. In this embodiment, the advertisement content is an advertisement image. This advertisement content is a virtual advertisement displayed in a virtual viewpoint image. The advertisement data acquisition unit 206 can acquire image data of the advertisement image from the user. The advertisement data acquisition unit 206 sends the acquired image data to the advertisement image generation unit 207. However, the type of advertisement content is not particularly limited. For example, the advertisement content may be a moving image or a three-dimensional model.

[0040] The advertisement image generation unit 207 and the advertisement synthesis unit 208 generate a virtual viewpoint image according to viewpoint information for a virtual space in which advertisement content designated in an advertisement area is placed. In this embodiment, the advertisement image generation unit 207 and the advertisement synthesis unit 208 generate a virtual viewpoint image by synthesizing advertisement content in an area of ​​a virtual viewpoint image generated in advance, the area corresponding to the advertisement area. At this time, the advertisement synthesis unit 208 can synthesize advertisement content in an area of ​​the virtual viewpoint image corresponding to the advertisement area, further based on depth information about the virtual viewpoint image.

[0041] In this embodiment, the advertising image generation unit 207 generates a virtual viewpoint image of the advertising content. Then, the advertising synthesis unit 208 synthesizes the virtual viewpoint image acquired by the image acquisition unit 201 and the virtual viewpoint image generated by the advertising image generation unit 207. Hereinafter, the virtual viewpoint image generated by the advertising image generation unit 207 will be referred to as an advertising virtual viewpoint image.

[0042] Specifically, the advertising image generation unit 207 generates a virtual viewpoint image of the advertising content arranged in the advertising area in the virtual space from a virtual viewpoint according to the parameters. First, the advertising image generation unit 207 arranges the advertising content acquired by the advertising data acquisition unit 206 in the advertising area in the virtual space according to the area information acquired by the area information acquisition unit 202. For example, the advertising image generation unit 207 can attach the advertising image acquired by the advertising data acquisition unit 206 to a virtual signboard according to the signboard information acquired by the area information acquisition unit 202. Then, the advertising image generation unit 207 generates a virtual viewpoint image of the advertising content according to the viewpoint information acquired by the viewpoint acquisition unit 203.

[0043] For example, the advertising image generation unit 207 can generate a virtual viewpoint image of a virtual signboard on which an advertising image is pasted, taken from a virtual camera, using the virtual camera parameters acquired from the viewpoint acquisition unit 203. Here, the advertising image generation unit 207 can generate depth information of the virtual viewpoint image. As already described, the virtual viewpoint image may have a depth channel representing such depth information. In this case, each pixel corresponding to the advertising image can have information about its distance from the virtual camera. Furthermore, each pixel not corresponding to the advertising image can have information representing infinity. The maximum distance value that a pixel can have may be used instead of infinity. Note that if the advertising data acquisition unit 206 has not acquired a designation of advertising content to be placed in the advertising area, the advertising image does not need to be displayed.

[0044] The advertisement synthesis unit 208 generates a virtual viewpoint image by synthesizing a virtual viewpoint image of advertising content arranged in an advertising area in a virtual space from a virtual viewpoint according to parameters with a previously generated virtual viewpoint image. Specifically, the advertisement synthesis unit 208 can generate a new virtual viewpoint image by synthesizing an input virtual viewpoint image acquired by the image acquisition unit 201 with an advertising virtual viewpoint image generated by the advertisement image generation unit 207. This new virtual viewpoint image corresponds to an input virtual viewpoint image including a new virtual advertisement or an input virtual viewpoint image including a changed virtual advertisement. Hereinafter, the virtual viewpoint image generated by the advertisement synthesis unit 208 is referred to as an output virtual viewpoint image. When generating an output virtual viewpoint video, the advertisement synthesis unit 208 can synthesize the input virtual viewpoint video and the advertising virtual viewpoint video for each frame.

[0045] Here, the advertisement synthesizer 208 may synthesize a virtual viewpoint image based on depth information. As described above, depth information corresponding to each of the input virtual viewpoint image and the advertisement virtual viewpoint image may exist. In this case, the advertisement synthesizer 208 can generate an output virtual viewpoint image by superimposing the advertisement virtual viewpoint image and the input virtual viewpoint image based on the depth information at each position of the advertisement virtual viewpoint image and the depth information at each position of the input virtual viewpoint image. The advertisement synthesizer 208 can superimpose the input virtual viewpoint image and the advertisement virtual viewpoint image pixel by pixel according to the depth value so that pixels with smaller depth values ​​are visible. In other words, the advertisement synthesizer 208 can synthesize the input virtual viewpoint image and the advertisement virtual viewpoint image so that pixels with a shallower depth from the virtual camera, i.e., pixels with a closer virtual distance from the virtual camera, are displayed.

[0046] The image output unit 209 outputs the output virtual viewpoint image generated by the advertisement synthesis unit 208 .

[0047] Next, an image processing method performed by the image processing system 100 according to one embodiment will be described with reference to the flowchart of FIG. 3 . In the following example, a user changes the advertising image used for a virtual advertisement. At this time, the image processing system 100 can calculate the advertising effectiveness of the virtual advertisement by the process shown in FIG. 3 . An example of calculating the advertising effectiveness of a virtual advertisement in a sports replay video will be described below. In the following example, the virtual viewpoint image is a video of a baseball game. However, the type of virtual viewpoint image is not particularly limited. For example, the virtual viewpoint image may be an image of another sport, such as a ball game such as soccer or basketball, or a martial art such as karate or sumo. The virtual viewpoint image may also be an image of a stage where a concert or a play is being held. In this way, the method of this embodiment can be applied to any content for which a virtual viewpoint image can be generated.

[0048] In S301, the image acquisition unit 201 acquires a virtual viewpoint image. In this example, the virtual viewpoint image is a replay video of a sports event.

[0049] In S302, the area information acquisition unit 202 acquires area information for the input virtual viewpoint image acquired in S301. FIG. 4 schematically shows a virtual signboard that is set in advance in association with the virtual viewpoint image. In FIG. 4, a stadium 401 is shown in the virtual viewpoint image. This stadium 401 is a three-dimensional model placed in a virtual space, simulating the stadium that is the subject of the image capture. Virtual advertisements 402a, 402b, and 402c are placed on the spectator stands of the stadium 401. Three virtual billboards are virtually placed on the spectator stands. Virtual advertisements 402a to 402c are pasted on the entire surface of each virtual billboard. The area information acquisition unit 202 acquires placement information for such virtual billboards, such as their position and size in the virtual space. In this way, multiple advertising areas may be placed in the virtual space. The area information acquisition unit 202 can acquire area information for each of the multiple advertising areas.

[0050] In S303, the viewpoint acquisition unit 203 acquires viewpoint information. The viewpoint acquisition unit 203 can acquire virtual camera parameters for the input virtual viewpoint image acquired in S301. FIG. 5 schematically shows the input virtual viewpoint image and the virtual camera parameters acquired in S303. This virtual viewpoint image corresponds to a home run replay scene. In this scene, the virtual camera is stationary so as to obtain a virtual viewpoint image according to the composition shown in FIG. 5. From this stationary position, the virtual camera captures the ball 501 flying into the stands.

[0051] In the following example, the input virtual viewpoint image is a 30-second video. The input virtual viewpoint image does not have to be a virtual viewpoint image obtained with the virtual camera's angle of view fixed. In other words, the angle of view of the virtual camera can be set in any way. The virtual camera can also move freely.

[0052] In S304, the advertising effect calculation unit 204 estimates the advertising effect. The advertising effect calculation unit 204 can calculate the advertising effect using the area information acquired by the area information acquisition unit 202 and the viewpoint information acquired by the viewpoint acquisition unit 203. As an example, a method for calculating the advertising effect for the virtual advertisement 402b will be described. A value indicating the advertising effect can be expressed, for example, according to the following formula (A). Advertising effectiveness = size x tilt x display position x time ……Formula (A)

[0053] Each parameter included in formula (A) will be explained. The size is a parameter corresponding to the size of the advertising area in the virtual viewpoint image where the virtual advertisement 402b is placed. The larger the advertising area, the larger this parameter can be. For example, the parameter corresponding to the size can be set according to the ratio of the area of ​​the virtual advertisement 402b to the area of ​​the virtual viewpoint screen. In this example, the size of the virtual advertisement 402b on the virtual viewpoint screen is 2%. In this case, the parameter corresponding to the size is 2.

[0054] The tilt is a parameter corresponding to the tilt of the advertising area in the virtual viewpoint image where the virtual advertisement 402b is placed. The smaller the tilt of the advertising area, the larger this parameter can be. The tilt-related parameter can be set, for example, based on the angle between the line extending from the virtual camera toward the virtual advertisement 402b and the normal to the virtual advertisement 402b. In this case, one of a plurality of pre-prepared parameters may be selected depending on the calculated angle. For example, the angle can be classified as maximum, large, medium, or small. If the virtual advertisement 402b is inverted, the angle may be classified as maximum. Then, a parameter corresponding to each classification can be used. For example, the tilt-related parameters may be maximum = 1 / 4, large = 1 / 3, medium = 1 / 2, and small = 2 / 3. In the example of FIG. 4, the angle of the virtual advertisement 402b can be classified as medium. In this case, the tilt-related parameter is 1 / 2. However, the tilt evaluation method is not limited to the above method. For example, a parameter according to the inclination may be set based on the angle between the optical axis direction vector of the virtual camera and the normal direction vector of virtual advertisement 402b.

[0055] The display position is a parameter corresponding to the position of the advertising area in the virtual viewpoint image where the virtual advertisement 402b is placed. The closer the advertising area is to the center of the virtual viewpoint image, the larger this parameter can be. For example, the position of the virtual advertisement 402b can be classified into a central portion and an edge portion. Then, a parameter corresponding to each classification can be used. For example, the parameter corresponding to the central portion may be 1, and the parameter corresponding to the edge portion may be 1 / 2. In the example of FIG. 4, the virtual advertisement 402b is displayed near the center of the virtual viewpoint image, so the parameter corresponding to the display position is 1.

[0056] The time is a parameter corresponding to the time that the advertising area in which the virtual advertisement 402b is placed is displayed in the virtual viewpoint image. The longer the time that the advertising area is displayed, the larger this parameter can be. For example, the number of seconds that the virtual advertisement 402b is displayed in the virtual viewpoint image can be used as the time-dependent parameter. In this example, the virtual advertisement 402b is constantly displayed in the virtual viewpoint image for 30 seconds. In this case, the time-dependent parameter is 30.

[0057] Based on the above parameters and calculation according to formula (A), the advertising effectiveness of virtual advertisement 402b is 30 as follows: 2×1 / 2×1×30=30

[0058] Note that the above-described method for calculating the advertising effectiveness is merely an example. For example, some of the parameters may be omitted or changed, or other parameters may be added. For example, the advertising effectiveness may be calculated based on the movement speed on the virtual viewpoint image of the advertising area where the virtual advertisement 402b is placed. In this case, the advertising effectiveness can be reduced as the movement speed on the virtual viewpoint image decreases.

[0059] Furthermore, the calculation method for each parameter is not limited to the above method, and other methods can be used. For example, a parameter corresponding to the tilt may be calculated from the tilt angle of the virtual advertisement 402b relative to the virtual camera using a function such as a trigonometric function. For example, if the virtual advertisement 402b is tilted 30 degrees relative to the virtual camera, cos30° may be used as the parameter. Furthermore, a parameter corresponding to the display position may be calculated according to the distance from the advertisement area to the center (or edge) of the virtual viewpoint image. In this case, the parameter corresponding to the display position may continuously change according to the position of the virtual advertisement 402b from the center (parameter = 1) to the edge (parameter = 1 / 2). Furthermore, if the position of the virtual viewpoint changes over time, the above-described calculation of the advertising effect can be performed for each frame of the virtual viewpoint image. If a one-second video consists of 60 frames, 1 / 60 can be used as the time parameter when calculating the advertising effect for each frame. Then, the sum of the advertising effects calculated for each frame can be calculated as the advertising effect of the virtual viewpoint image. In this way, any method can be used to calculate the advertising effect without departing from the spirit of the present invention.

[0060] In S305, the user management unit 205 authenticates that the user using the image processing system 100 has the right to place a virtual advertisement in the input virtual viewpoint image. The user management unit 205 can determine, through user authentication, whether the user is a user who can change the virtual advertisement for the input virtual viewpoint image. If the user management unit 205 determines that the user has the right to change the virtual advertisement, the process proceeds to S306. If not, the process proceeds to S308.

[0061] In S306, the display control unit 210 presents to the user information indicating the advertising effect estimated by the advertising effect calculation unit 204 for the advertising area. Here, the advertising data acquisition unit 206 can acquire from the user a designation of advertising content to be placed in the advertising area. The advertising data acquisition unit 206 determines whether or not advertising content has been designated. If the advertising data acquisition unit 206 determines that advertising content has been designated, the process proceeds to S307. If not, the process proceeds to S308.

[0062] In S307, the advertisement data acquisition unit 206 acquires data of the specified advertisement content. In this embodiment, the advertisement data acquisition unit 206 acquires image data of the virtual advertisement. The image processing system 100 may manage the data of the advertisement content separately from the virtual viewpoint image. In this case, the advertisement data acquisition unit 206 can change the image data of the virtual advertisement, which is managed in association with the virtual viewpoint image, to the image data acquired in S307.

[0063] The display control unit 210 can generate a UI including a virtual viewpoint image and information indicating an estimated advertising effectiveness for a preset advertising area associated with the virtual viewpoint image. This UI can further include an element operated to specify advertising content to be placed in the advertising area. Figures 6(A) to 6(D) schematically show examples of such a UI including an operation screen used to change the virtual advertisement. However, the configuration of the operation screen is not limited to the following example.

[0064] Window 600 shown in FIG. 6(A) includes a login screen. A user who wishes to change a virtual advertisement logs in by entering a user ID and password in window 600. At this time, user authentication is performed by user management unit 205. The method of user authentication is not particularly limited. For example, identity verification using biometric authentication, authentication using a public encryption key method such as RSA encryption, or authentication using a hardware key may be performed.

[0065] Window 601 shown in FIG. 6(B) includes an operation screen for selecting a virtual viewpoint image for virtual advertising. Video area 602 displays live video of the current game. In this example, video area 602 displays a virtual viewpoint image generated in real time. Such a virtual viewpoint image may be generated by image processing system 100 or another system. However, video area 602 may also display an image captured using a normal camera.

[0066] A list of virtual viewpoint images is displayed in the list box 603. The list box 603 displays virtual viewpoint images as archived footage. As the match progresses, generated virtual viewpoint images are appropriately registered in the list box 603. Such virtual viewpoint images may be generated by the image processing system 100 or by another system. The registered virtual viewpoint images may be live footage from a specific time segment that was previously displayed in the video area 602. The registered virtual viewpoint images may also be virtual viewpoint images generated based on instructions from a creator. In this case, the creator can set a virtual viewpoint to be used to generate the virtual viewpoint images. The creator may be, for example, a video distributor such as a television station. The image acquisition unit 201 can acquire the virtual viewpoint images registered in this manner.

[0067] In the example of FIG. 6(B), scenes 604a to 604c are registered as virtual viewpoint images, which are archived videos. In the case of video of a baseball game, examples of archived videos to be registered include scenes of scoring, spectacular scenes such as fine defensive plays, unexpected scenes, and scenes of players achieving records. Archived videos may also be videos shot using standard camerawork. On the other hand, archived videos may also be videos viewed from the field during a game, videos from the viewpoint of a specific player such as the pitcher or catcher, or videos of chasing a ball, which are difficult to create using standard cameras. Such videos from viewpoints specific to virtual viewpoint images can be registered as archived videos.

[0068] Icons 605a to 605c have the function of transitioning to a screen for editing advertising images. The user can select this icon to transition to an advertising image editing screen for the scene corresponding to each icon. In this example, selecting each of icons 605a to 605c transitions to an advertising image editing screen for scenes 604a to 604c. For example, selecting icon 605a displays advertising image editing screen 606 for scene 604a. In this example, scene 604a is the home run replay scene shown in FIG. 5. The virtual viewpoint image of scene 604a includes virtual advertisements 402a to 402c.

[0069] 6(C) is a screen for specifying an advertisement image. On the advertisement image editing screen 606, information about the advertisement and the advertisement image can be edited. The advertisement image editing screen 606 has a virtual viewpoint image of the scene 604a.

[0070] The virtual advertisement list 607 is a list of virtual advertisements included in the scene 604a. The user can specify a virtual advertisement to be edited in the virtual advertisement list 607. In this example, a plurality of advertisement areas are set in advance in association with the virtual viewpoint image. In addition, the virtual advertisement 402b, which is a virtual advertisement to be placed in one of the plurality of advertisement areas, is selected.

[0071] The advertisement information 608 is an area where information about the virtual advertisement is displayed. The advertisement information 608 displays information corresponding to the virtual advertisement selected in the virtual advertisement list 607. The advertisement information 608 displays the advertising effectiveness estimated by the advertising effectiveness calculation unit 204 for the virtual advertisement. Thus, the advertisement information 608 displays information indicating the advertising effectiveness estimated for an advertising area previously set in association with the virtual viewpoint image of the scene 604a. In this embodiment, the advertisement information 608 displays the advertising effectiveness estimated for an advertising area selected from multiple advertising areas. Alternatively, the advertisement image editing screen 606 may display the advertising effectiveness estimated for each of the multiple advertising areas. In the example of FIG. 6(C), the advertisement information 608 displays the numerical value of the advertising effectiveness, as well as the display position, display time, display size, and display angle of the advertisement. The advertisement information 608 also displays statistical information regarding the playback of the virtual viewpoint image of the scene 604a. In this example, the statistical information includes the number of plays per unit time just before and the total number of plays. In this example, one minute is set as the unit time.

[0072] The user can set the advertising content of the virtual advertisement 402b while referring to the advertising effectiveness and the number of views. For example, for an advertising area with high advertising effectiveness, a more important advertisement such as announcing the release of a new product can be set. For an advertising area with low advertising effectiveness, an advertising image showing a standard product or a brand logo can be set.

[0073] The advertisement image editing screen 606 also includes an image of the advertisement content currently designated to be placed in the advertisement area. In the example of FIG. 6(C), the advertisement image editing screen 606 includes an advertisement image 609 set as the virtual advertisement 402b. The advertisement image editing screen 606 further includes an advertisement image change button 610 as an element operated to designate the advertisement content to be placed in the advertisement area. The advertisement image change button 610 is a button for performing an operation to change the advertisement image for the virtual advertisement selected in the virtual advertisement list 607. The advertisement image displayed in the virtual advertisement can be changed by selecting the advertisement image change button 610 and designating a new virtual advertisement image.

[0074] In S308, the advertising image generation unit 207 generates an advertising virtual viewpoint image for the advertising content specified in S306. The advertising image generation unit 207 generates a virtual viewpoint image of a virtual advertisement from the same virtual camera as the virtual viewpoint image acquired in S301, using the advertising image specified in S306, the area information acquired in S302, and the viewpoint information acquired in S303.

[0075] In S309, the advertisement synthesis unit 208 synthesizes the input virtual viewpoint image acquired in S301 with the advertisement virtual viewpoint image generated in S307. FIG. 7 is a schematic diagram showing a method for synthesizing the input virtual viewpoint image with the advertisement virtual viewpoint image. FIG. 7(A) is a schematic diagram showing the input virtual viewpoint image for the scene 604a acquired by the image acquisition unit 201. In this example, no virtual advertisement is synthesized in the virtual viewpoint image acquired by the image acquisition unit 201. However, the image acquisition unit 201 may also acquire a virtual viewpoint image with a virtual advertisement synthesized therein (for example, a virtual viewpoint image previously generated by the advertisement synthesis unit 208).

[0076] 7(B) is a schematic diagram showing an advertisement virtual viewpoint image generated by the advertisement image generation unit 207. The viewpoint acquisition unit 203 acquires the same virtual camera parameters as those of the input virtual viewpoint image for the scene 604a. The advertisement image generation unit 207 can generate an advertisement virtual viewpoint image by rendering a virtual advertisement 402b using the virtual camera parameters acquired by the viewpoint acquisition unit 203. In this example, the virtual advertisement 402b is rendered according to the advertisement image specified in S307.

[0077] 7(C) is a schematic diagram of an output virtual viewpoint image generated by the advertisement synthesis unit 208 by synthesizing the input virtual viewpoint image shown in FIG. 7(A) and the virtual advertisement image shown in FIG. 7(B). In this example, a virtual advertisement 402b is present on the stands of the stadium 401 in the virtual viewpoint image shown in FIG. 7(A). The virtual advertisement 402b is closer to the virtual camera than the stadium 401. Therefore, the depth of the virtual advertisement 402b from the virtual camera is shallower than that of the stadium 401. For this reason, the virtual advertisement 402b is synthesized so as to be superimposed on the stadium 401.

[0078] In S310, the image output unit 209 outputs the output virtual viewpoint image generated by the advertisement synthesis unit 208. The image output unit 209 may store the output virtual viewpoint image in a memory within the image processing system 100, or may output the output virtual viewpoint image to an external device such as a server. The archived video shown in FIGS. 6(B) and (C) (e.g., the archived video for scene 604a) can be updated to reflect the virtual viewpoint image output by the image output unit 209. Furthermore, the image processing system 100 or an external device such as a server can distribute the virtual viewpoint image thus generated to users other than the advertiser.

[0079] According to the above embodiments, an advertiser can set a virtual advertisement taking into consideration the advertising effectiveness of the virtual advertisement estimated when the virtual advertisement is placed in an advertising area. Such technology can support the advertiser in selecting an advertisement. Furthermore, according to the above embodiments, the virtual advertisement can be changed depending on the advertising effectiveness, making it easier for the advertiser to run an optimal advertisement. Furthermore, according to the above embodiments, it is possible to set a virtual advertisement taking into consideration the advertising effectiveness without repeatedly selecting a virtual advertisement and generating a virtual viewpoint image in which the virtual advertisement is placed. This reduces the operational burden on the advertiser, as well as the processing load and power consumption of the system.

[0080] The above embodiment is merely an example. For example, the output virtual viewpoint image may be generated by a different method. For example, the advertisement synthesis unit 208 may generate the virtual viewpoint image using a three-dimensional model of a virtual object arranged in the virtual space, advertisement content arranged in an advertisement area in the virtual space, and viewpoint information.

[0081] Specifically, the image acquisition unit 201 may acquire a three-dimensional model (e.g., three-dimensional shape data and color information) of a virtual object placed in the virtual space. In this case, the advertisement synthesis unit 208 can generate an output virtual viewpoint image using the three-dimensional model of the virtual object placed in the virtual space, the advertising content placed in the advertising area in the virtual space, and viewpoint information. For example, the advertisement image generation unit 207 can generate a three-dimensional model of the advertising content (e.g., three-dimensional shape data and color information) that indicates the advertising content placed in the advertising area. Furthermore, the advertisement synthesis unit 208 synthesizes the three-dimensional model of the virtual object with the three-dimensional model of the advertising content. Then, the advertisement synthesis unit 208 can render a virtual viewpoint image of the virtual space in which these three-dimensional models are placed, according to the virtual camera parameters acquired by the viewpoint acquisition unit 203. In this manner, the advertisement synthesis unit 208 can newly generate a virtual viewpoint image including a virtual advertisement.

[0082] Furthermore, the user management unit 205 can manage whether or not a user has the right to place and post a virtual advertisement in an advertisement area shown in a virtual viewpoint image. As shown in Fig. 6(B), a plurality of advertisement areas may be set in association with one virtual viewpoint image. In this case, the user management unit 205 may manage whether or not a user has the right to place and post a virtual advertisement for each of a plurality of advertisement areas associated with one virtual viewpoint image.

[0083] (Transfer of rights to place virtual advertisements) Furthermore, the right to place a virtual advertisement in a specific advertising area may be transferable between users. In this case, the user management unit 205 can manage the right so that the right can be transferred between users. In such a modified example, the image processing system 100 may further include a distribution processing unit 212. The distribution processing unit 212 can perform transfer processing between users regarding the right to place a virtual advertisement. Furthermore, the distribution processing unit 212 can perform contract processing such as buying and selling or transferring the right between users. Furthermore, the distribution processing unit 212 can instruct the user management unit 205 to transfer the right between users when a contract is concluded. The contract processing between users is not particularly limited. For example, the contract processing may be realized through online commerce.

[0084] 8(A) to 8(C) schematically show examples of UIs including operation screens used to buy and sell virtual advertisements. FIG. 8(A) shows a screen for selecting a scene for purchasing the right to place a virtual advertisement. When a user presses an advertisement purchase button 801, the screen transitions to a purchase screen for the right to place a virtual advertisement in the corresponding scene 604a. A user who wants to purchase the right to place a virtual advertisement in scene 604a selects the advertisement purchase button 801. When the advertisement purchase button 801 is pressed, the screen shown in FIG. 8(B) is displayed.

[0085] The screen shown in Fig. 8(B) displays, similarly to Fig. 6(C), a virtual viewpoint image of the selected scene 604a, a virtual advertisement list 607, and advertisement information 608 including advertisement effectiveness. Also, advertisement price 802 is an area that displays information regarding the price of the right to place a virtual advertisement. Fig. 8(B) displays the current price of the right.

[0086] The current price may be a selling price desired by a user who owns the virtual advertisement. The distribution processing unit 212 may also calculate the current price for the right to place a virtual advertisement in an advertising area based on the advertising effectiveness estimated for this advertising area. The distribution processing unit 212 may also calculate the current price based on statistical information regarding the playback of virtual viewpoint images. For example, the distribution processing unit 212 can predict the future number of plays of the virtual viewpoint image based on the trend in the number of plays per unit time. The distribution processing unit 212 can then calculate the current price according to the predicted number of plays and advertising effectiveness. For example, if the number of plays per unit time is on the rise, the distribution processing unit 212 can predict that the future number of plays will increase. In this case, the price of the rights will rise. On the other hand, if the number of plays per unit time is on the decline, the distribution processing unit 212 can predict that the future number of plays will decrease. In this case, the price of the rights will fall. In this way, the current price of the rights may fluctuate. Immediately after the virtual viewpoint image of scene 604a is registered, the distribution processing unit 212 may calculate the current price based on statistical information regarding the playback of other virtual viewpoint images, such as the average value of the total number of times that virtual viewpoint images have been played in the past.

[0087] The advertisement purchase request button 803 is a button that is pressed to request the purchase of the right to place a virtual advertisement in the advertisement area. The user can determine whether or not to make a request to purchase the rights by checking the advertising effectiveness or advertising price 802 shown in the advertisement information 608. A user who wants to purchase the rights can make an advertisement purchase request by inputting a desired purchase price 804 and pressing the advertisement purchase request button.

[0088] In this embodiment, when the advertisement purchase request button is pressed, a notification is sent to the user who owns the requested rights. This notification can be sent, for example, by the display control unit 210 displaying a window shown in FIG. 8(C) on the screen. This window includes a message 805 describing the purchase conditions for the rights (e.g., the desired purchase price). The display control unit 210 may also present the advertising effectiveness of the requested advertising area or the current price to the user who owns the rights. For example, the display control unit 210 may display this information in the window shown in FIG. 8(C). The user who owns the rights refers to the message 805 to determine whether or not to allow the sale of the rights. If the user wishes to sell the rights, the user presses a Yes button 806. In this case, the distribution processing unit 212 can determine that a sales contract has been established. If the user does not wish to sell the rights, the user presses a No button 807.

[0089] 9 is a flowchart of the process performed by the image processing system 100 for trading the right to place a virtual advertisement in an advertisement area between users. In S901, a user who wants to purchase the right to place a virtual advertisement in an advertisement area issues a purchase request for the right to a user who has this right. The request can be made using a UI such as that shown in FIG. 8(A).

[0090] In S902, the distribution processing unit 212 determines whether the sale of the rights has been established. For example, if the user who owns the rights has permitted the sale of the rights (for example, if the user presses the Yes button 806), the distribution processing unit 212 determines that the sale of the rights has been established. If the distribution processing unit 212 determines that the sale of the rights has been established, the process proceeds to S903. If not, the process ends.

[0091] In S903, the user management unit 205 performs a process of transferring the right to place a virtual advertisement in the advertisement area from the user who sold the right to the user who purchased the right. In S904, the user who has acquired the right to place a virtual advertisement in the advertisement area performs a process of changing the virtual advertisement to be placed in this advertisement area. The process of changing the virtual advertisement can be executed according to the flowchart shown in FIG.

[0092] According to the above configuration, the right to place a virtual advertisement in a virtual viewpoint image can be traded and distributed. Furthermore, a user can refer to the estimated advertising effect of the virtual advertisement to determine whether to trade the right.

[0093] The method for managing the right to place a virtual advertisement is not particularly limited. For example, the right to place a virtual advertisement may be managed using a blockchain. Specifically, the user management unit 205 can assign a unique identification ID to the right to place a virtual advertisement in a specific advertising area of ​​a specific virtual viewpoint image. The user management unit 205 can then manage the ownership of the virtual advertisement by registering an NFT indicating the ownership of such a right on the blockchain. In this way, the user management unit 205 can manage the rights using the blockchain. Furthermore, the contract processing performed by the distribution processing unit 212 may be executed using smart contract technology realized by the blockchain.

[0094] When rights are managed using NFTs, the user management unit 205 may perform user authentication by comparing an identification ID indicated by an NFT owned by the user with a unique identification ID assigned to the right to place a virtual advertisement in an advertising area. The type of identification ID is not limited. For example, a hash value may be used as the identification ID. Also, a unique ID may be assigned to each of the generated virtual viewpoint image and the advertising area set in the virtual viewpoint image. In this case, these IDs may be combined to generate a unique identification ID assigned to the right. Also, such a unique identification ID may be included in the signboard information acquired by the area information acquisition unit 202.

[0095] In the above embodiment, a user who has the right to place a virtual advertisement in an advertising area determines whether to sell the right in response to a purchase request from another user. However, other embodiments are also possible. For example, a user may set a desire to purchase the right without specifying a target virtual viewpoint image or advertising area. In response to such a desire, a user who owns the right can input a proposal for a right that can be sold. A UI screen including a list of proposed saleable rights is displayed to a user who wishes to purchase the right. The user can select the right that they wish to purchase from the list and make a purchase request. The user who owns the right may also set a desired selling price. In this case, if the user makes a purchase request for a price higher than the set selling price, the sale is automatically permitted. The right may also be bought and sold in an auction format. For example, a user who wishes to sell the right can set an end time. Then, the right can be automatically sold to the user who makes a purchase request for the highest price by the end time. The user may also set in advance not to sell the right that they own. In this case, when a purchase request is received, the sale can be automatically refused. If another person has the right to place a virtual advertisement in the advertising area and such a setting is made, the advertisement purchase request button 803 shown in Figure 8(B) or the items in the virtual advertisement list 607 may be unselectable.

[0096] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The disclosure of this specification includes the following image processing system, image processing method, and program. (Item 1) a viewpoint acquisition means for acquiring information about a virtual viewpoint set in a virtual space; an area acquisition means for acquiring area information indicating an advertising area arranged in the virtual space; an estimation means for estimating an advertising effectiveness for the advertising area based on the virtual viewpoint information and the area information; a presentation means for presenting information indicating the estimated advertising effectiveness for the advertising area; an advertisement acquisition means for acquiring, after presenting information indicating the estimated advertising effectiveness for the advertising area, a designation of advertising content to be placed in the advertising area; a generating means for generating a virtual viewpoint image according to the information on the virtual viewpoint for the virtual space in which the advertising content designated in the advertising area is placed; An image processing system comprising: (Item 2) 2. The image processing system according to item 1, wherein the area information is set in advance in association with a virtual viewpoint image that has been generated in advance. (Item 3) 3. The image processing system according to any one of items 1 to 2, wherein the virtual viewpoint information represents a virtual viewpoint parameter for a virtual viewpoint image that has been generated in advance. (Item 4) Item 3. The image processing system according to item 3, wherein the generating means generates the virtual viewpoint image by synthesizing the advertising content in an area of ​​the pre-generated virtual viewpoint image corresponding to the advertising area. (Item 5) The image processing system described in item 4, characterized in that the generation means synthesizes the advertising content in an area corresponding to the advertising area of ​​the virtual viewpoint image further based on depth information about the virtual viewpoint image. (Item 6) The image processing system according to any one of items 3 to 5, wherein the generating means generates the virtual viewpoint image by synthesizing a virtual viewpoint image of the advertising content placed in the advertising area in the virtual space from a virtual viewpoint according to the parameters with the pre-generated virtual viewpoint image. (Item 7) The image processing system described in item 6 is characterized in that the generation means generates the virtual viewpoint image by superimposing the virtual viewpoint image of the advertising content and the pre-generated virtual viewpoint image based on depth information of each position of the virtual viewpoint image of the advertising content and depth information of each position of the pre-generated virtual viewpoint image. (Item 8) 8. The image processing system according to any one of items 1 to 7, wherein the generating means generates the virtual viewpoint image using a three-dimensional model of a virtual object placed in the virtual space, the advertising content placed in the advertising area in the virtual space, and information about the virtual viewpoint. (Item 9) The image processing system according to any one of items 1 to 8, characterized in that the estimation means estimates the advertising effectiveness of the advertising area based on at least one of the size and tilt of the advertising area shown in a virtual viewpoint image of the virtual space according to the information of the virtual viewpoint. (Item 10) the virtual viewpoint information represents a virtual viewpoint that changes over time; The image processing system according to any one of items 1 to 9, characterized in that the estimation means estimates the advertising effectiveness for the advertising area based on at least one of the display time of the advertising area in the virtual viewpoint image of the virtual space according to the information of the virtual viewpoint and the movement speed of the advertising area. (Item 11) The image processing system according to any one of items 1 to 10, wherein the estimation means estimates the advertising effectiveness based on at least one of a resolution of a virtual viewpoint image, a compression rate of the virtual viewpoint image, and a frame rate of a virtual viewpoint video, which are generated for the virtual space in which the advertising content is placed in the advertising area. (Item 12) The method further includes: authenticating means for authenticating that a user has the right to place advertising content in the advertising area; 12. The image processing system according to any one of items 1 to 11, wherein the advertisement acquisition means receives a designation of advertisement content to be placed in the advertisement area from the user when the user is authenticated as having the right. (Item 13) Item 13. The image processing system according to item 12, further comprising a management means for managing the rights so that the rights can be transferred between users. (Item 14) Item 14. The image processing system according to item 13, wherein the management means manages the rights using a blockchain. (Item 15) 15. The image processing system according to any one of items 1 to 14, further comprising a display control unit that generates a user interface including a virtual viewpoint image, information indicating the estimated advertising effectiveness for an advertising area that is preset in association with the virtual viewpoint image, and an element that is operated to specify advertising content to be placed in the advertising area. (Item 16) A plurality of advertising areas are set in advance in association with the virtual viewpoint image, Item 16. The image processing system of item 15, wherein the user interface includes information indicating the estimated advertising effectiveness for each of the plurality of advertising areas or for an advertising area selected from the plurality of advertising areas. (Item 17) Item 17. The image processing system of item 16, wherein the user interface includes an image of advertising content currently designated to be placed in the advertising area. (Item 18) 18. The image processing system according to any one of items 15 to 17, wherein the user interface includes an element that can be operated to purchase the right to place advertising content in the advertising area. (Item 19) An image processing method performed by an image processing system, acquiring information about a virtual viewpoint set in a virtual space; acquiring area information indicating an advertising area arranged in the virtual space; estimating an advertising effect for the advertising area based on the virtual viewpoint information and the area information; presenting information indicating the estimated advertising effectiveness for the advertising area; obtaining a designation of advertising content to be placed in the advertising area after presenting information indicating the estimated advertising effectiveness for the advertising area; generating a virtual viewpoint image according to the virtual viewpoint information for the virtual space in which the advertising content designated in the advertising area is placed; An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing system according to any one of items 1 to 18.

[0098] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0099] 201: Image acquisition unit, 202: Area information acquisition unit, 203: Viewpoint acquisition unit, 204: Advertising effect calculation unit, 205: User management unit, 206: Advertising data acquisition unit, 207: Advertising image generation unit, 208: Advertising synthesis unit, 209: Image output unit, 210: Display control unit

Claims

1. a viewpoint acquisition means for acquiring information about a virtual viewpoint set in a virtual space; an area acquisition means for acquiring area information indicating an advertising area arranged in the virtual space; an estimation means for estimating an advertising effectiveness for the advertising area based on the virtual viewpoint information and the area information; a presentation means for presenting information indicating the estimated advertising effectiveness for the advertising area; an advertisement acquisition means for acquiring, after presenting information indicating the estimated advertising effectiveness for the advertising area, a designation of advertising content to be placed in the advertising area; a generating means for generating a virtual viewpoint image according to the information on the virtual viewpoint for the virtual space in which the advertising content designated in the advertising area is placed; An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the area information is set in advance in association with a virtual viewpoint image that has been generated in advance.

3. 2. The image processing system according to claim 1, wherein the virtual viewpoint information represents a virtual viewpoint parameter for a virtual viewpoint image that has been generated in advance.

4. 4. The image processing system according to claim 3, wherein the generating means generates the virtual viewpoint image by synthesizing the advertising content in an area of ​​the pre-generated virtual viewpoint image that corresponds to the advertising area.

5. The image processing system according to claim 4 , wherein the generating means synthesizes the advertising content in an area of ​​the virtual viewpoint image corresponding to the advertising area, further based on depth information about the virtual viewpoint image.

6. The image processing system of claim 3, wherein the generation means generates the virtual viewpoint image by combining a virtual viewpoint image of the advertising content placed in the advertising area in the virtual space from a virtual viewpoint according to the parameters with the pre-generated virtual viewpoint image.

7. The image processing system according to claim 6, wherein the generation means generates the virtual viewpoint image by superimposing the virtual viewpoint image of the advertising content and the pre-generated virtual viewpoint image based on depth information of each position of the virtual viewpoint image of the advertising content and depth information of each position of the pre-generated virtual viewpoint image.

8. 2. The image processing system according to claim 1, wherein the generating means generates the virtual viewpoint image using a three-dimensional model of a virtual object arranged in the virtual space, the advertising content arranged in the advertising area in the virtual space, and information about the virtual viewpoint.

9. The image processing system of claim 1, wherein the estimation means estimates the advertising effectiveness for the advertising area based on at least one of the size and inclination of the advertising area shown in the virtual viewpoint image of the virtual space according to the virtual viewpoint information.

10. the virtual viewpoint information represents a virtual viewpoint that changes over time; The image processing system of claim 1, characterized in that the estimation means estimates the advertising effectiveness for the advertising area based on at least one of the display time of the advertising area in the virtual viewpoint image of the virtual space according to the virtual viewpoint information and the movement speed of the advertising area.

11. 2. The image processing system of claim 1, wherein the estimation means estimates the advertising effectiveness based on at least one of a resolution of a virtual viewpoint image, a compression rate of the virtual viewpoint image, and a frame rate of a virtual viewpoint video generated for the virtual space in which the advertising content is placed in the advertising area.

12. The method further includes: authenticating means for authenticating that a user has the right to place advertising content in the advertising area; 2. The image processing system according to claim 1, wherein the advertisement acquisition means accepts, from the user, a designation of advertisement content to be placed in the advertisement area when the user is authenticated as having the right.

13. 13. The image processing system according to claim 12, further comprising a management unit that manages the rights so that the rights can be transferred between users.

14. The image processing system according to claim 13 , wherein the management means manages the rights using a blockchain.

15. 2. The image processing system according to claim 1, further comprising a display control unit that generates a user interface including a virtual viewpoint image, information indicating the estimated advertising effectiveness for an advertising area that is preset in association with the virtual viewpoint image, and an element that is operated to specify advertising content to be placed in the advertising area.

16. A plurality of advertising areas are set in advance in association with the virtual viewpoint image, 16. The image processing system of claim 15, wherein the user interface includes information indicating the estimated advertising effectiveness for each of the plurality of advertising areas or for an advertising area selected from the plurality of advertising areas.

17. 17. The image processing system of claim 16, wherein the user interface includes an image of advertising content currently designated for placement in the advertising region.

18. 16. The image processing system according to claim 15, wherein the user interface includes an element that is operated to purchase the right to place advertising content in the advertising area.

19. An image processing method performed by an image processing system, acquiring information about a virtual viewpoint set in a virtual space; acquiring area information indicating an advertising area arranged in the virtual space; estimating an advertising effect for the advertising area based on the virtual viewpoint information and the area information; presenting information indicating the estimated advertising effectiveness for the advertising area; obtaining a designation of advertising content to be placed in the advertising area after presenting information indicating the estimated advertising effectiveness for the advertising area; generating a virtual viewpoint image according to the virtual viewpoint information for the virtual space in which the advertising content designated in the advertising area is placed; An image processing method comprising:

20. A program for causing a computer to function as the image processing system according to any one of claims 1 to 18.

Images