Apparatus and method for processing image and program

Abstract

To provide an apparatus and method for processing an image and program that allow a viewer who is enjoying a virtual-viewpoint image to see a particular image.SOLUTION: An apparatus for processing an image, in a system for processing an image 2, comprises a processor and a storage connected to or incorporated in the processor. The processor acquires a virtual-viewpoint image taken by an imaging device and generated based on a plurality of photographic images and outputs first data for a first particular image selected from a plurality of particular images to be displayed in a first area based on first information associated with a first area concerned with the virtual-viewpoint image and particular image-related information related to a plurality of particular images not included in the plurality of photographic images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] JP 2020-101897 A discloses an information processing device characterized by having an acquisition means for acquiring virtual viewpoint information indicating the position and direction of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images taken from different directions by a plurality of imaging devices, and a determination means for determining an amount according to the display mode of an advertising image displayed in an advertising space in the virtual viewpoint image corresponding to the virtual viewpoint indicated by the virtual viewpoint information, based on information indicating an advertising space previously set in a virtual space and the virtual viewpoint information.

[0003] International Publication No. 2016 / 194441 discloses a three-dimensional advertising space determination system comprising a user terminal that receives content data composed of free viewpoint videos from a distribution computer, which allows the viewpoint from which the viewing viewpoint can be changed, displays video data from a specific viewpoint on a display means, and, when an operator provides viewpoint characteristic change data that changes the viewpoint characteristics of the displayed specific viewpoint video data, displays specific viewpoint video data based on this viewpoint characteristic change data on the display means, and a three-dimensional advertising space determination computer, wherein A) the user terminal further comprises: an attention space determination means that determines an attention space in the specific viewpoint video based on the specific viewpoint video data displayed on the display means, and a transmission means that transmits a history of changes in the attention space to the three-dimensional advertising space determination computer; and B) the three-dimensional advertising space determination computer comprises: a user-specific history data receiving means that receives the history of changes in the content data, and a three-dimensional advertising space determination means that determines a three-dimensional advertising space for each content obtained from the user-specific history data.

[0004] JP 2020-101847 A discloses an image file generation device that generates an image file for generating a virtual viewpoint image, characterized in that it comprises a material information acquisition means for acquiring material information used to generate the virtual viewpoint image, an additional information acquisition means for acquiring additional information to be displayed on the virtual viewpoint image, and an image file generation means for generating an image file including the material information and the additional information. Summary of the Invention

[0005] One embodiment of the technique of the present disclosure provides an image processing device, an image processing method, and a program that can show a specific image to a viewer viewing a virtual viewpoint image. [Means for solving the problem]

[0006] A first aspect of the technology disclosed herein is an image processing device comprising a processor and a memory connected to or built into the processor, in which the processor acquires a virtual viewpoint image generated based on a plurality of captured images, and outputs first data for displaying a first specific image selected from a plurality of specific images in the first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images not included in the plurality of captured images.

[0007] A second aspect of the technology of the present disclosure is the image processing device according to the first aspect, in which the first information includes first content-related information related to the content of the virtual viewpoint image.

[0008] A third aspect of the technology disclosed herein is an image processing device according to the second aspect, in which the specific image related information includes second content related information related to the content of the specific image, and the first specific image is a specific image among multiple specific images that is related to specific image related information that includes second content related information corresponding to the first content related information.

[0009] A fourth aspect of the technology of the present disclosure is the image processing device according to any one of the first to third aspects, in which the first information includes first advertising effect related information relating to advertising effect.

[0010] A fifth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourth aspects, wherein the first information includes first size-related information regarding the size at which the first specific image is displayed in the first area.

[0011] A sixth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fifth aspects, wherein the first information includes first viewpoint information required to generate a virtual viewpoint image.

[0012] A seventh aspect of the technology of the present disclosure is the image processing device according to the sixth aspect, in which the first viewpoint information includes information related to a first viewpoint path.

[0013] An eighth aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, wherein the first information includes first display time related information related to the time at which the first area is displayed.

[0014] A ninth aspect of the technique of the present disclosure is the image processing device according to the eighth aspect, in which the first display time related information is information related to a time during which the first area is continuously displayed.

[0015] A tenth aspect of the technology of the present disclosure is an image processing device according to the eighth or ninth aspect, in which the specific image is a moving image, the specific image related information includes a total playback time of the moving image, and the processor generates first data based on the first display time related information and the total playback time.

[0016] An eleventh aspect of the technology of the present disclosure is an image processing device according to any one of the eighth to tenth aspects, in which the specific image is a moving image, the specific image related information includes a total playback time of the moving image, and the processor selects the first specific image based on the first display time related information and the total playback time.

[0017] A twelfth aspect of the technology of the present disclosure is an image processing device according to any one of the first to eleventh aspects, in which the virtual viewpoint image is a moving image and the first information includes first timing-related information related to the timing at which the first area is included in the virtual viewpoint image.

[0018] A thirteenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to twelfth aspects, wherein the first information includes first movement speed related information related to the movement speed of the first viewpoint required to generate a virtual viewpoint image.

[0019] A fourteenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to thirteenth aspects, in which the first information is changed depending on at least one of the viewpoint position, the line of sight direction, and the angle of view required to generate a virtual viewpoint image.

[0020] A 15th aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourteenth aspects, in which the processor further outputs second data for displaying a second specific image selected from a plurality of specific images in the second area based on second information associated with the second area related to the virtual viewpoint image and specific image related information.

[0021] A sixteenth aspect of the technology of the present disclosure is the image processing device according to the fifteenth aspect, in which the second information includes second advertising effect related information relating to advertising effect.

[0022] A 17th aspect of the technology of the present disclosure is an image processing device according to the 15th or 16th aspect, in which the second information includes second size-related information regarding the size at which the second specific image is displayed in the second area.

[0023] An 18th aspect of the technology of the present disclosure is an image processing device according to any one of the 15th to 17th aspects, in which the second information includes second viewpoint information required to generate a virtual viewpoint image.

[0024] A nineteenth aspect of the technology of the present disclosure is the image processing device according to the eighteenth aspect, in which the second viewpoint information includes information related to a second viewpoint path.

[0025] A twentieth aspect of the technology of the present disclosure is an image processing device according to any one of the fifteenth to nineteenth aspects, wherein the second information includes second display time related information related to the time at which the second area is displayed.

[0026] A 21st aspect of the technique of the present disclosure is the image processing device according to the 20th aspect, in which the second display time related information is information related to a time for which the second area is continuously displayed.

[0027] A 22nd aspect of the technology of the present disclosure is an image processing device according to the 20th aspect, in which the specific image is a moving image, the specific image related information includes a total playback time of the moving image, and the processor generates second data based on the second display time related information and the total playback time.

[0028] A 23rd aspect of the technology of the present disclosure is an image processing device relating to any one of the 20th to 22nd aspects, in which the specific image is a moving image, the specific image related information includes a total playback time of the moving image, and the processor selects a second specific image based on the second display time related information and the total playback time.

[0029] A 24th aspect of the technology of the present disclosure is an image processing device according to any one of the 15th to 23rd aspects, in which the virtual viewpoint image is a moving image and the second information includes second timing-related information related to the timing at which the second area is included in the virtual viewpoint image.

[0030] A 25th aspect of the technology disclosed herein is an image processing device according to any one of the 15th to 24th aspects, in which the second information includes second movement speed related information related to the movement speed of the second viewpoint required to generate the virtual viewpoint image.

[0031] A 26th aspect of the technology of the present disclosure is an image processing device according to any one of the 15th to 25th aspects, in which the second information is changed depending on at least one of the viewpoint position, line of sight direction, and angle of view required to generate a virtual viewpoint image.

[0032] A 27th aspect of the technology of the present disclosure is an image processing device according to any one of the first to 26th aspects, in which the specific image related information includes billing information of the party providing the specific image.

[0033] A 28th aspect of the technology of the present disclosure is an image processing device according to any one of the 1st to 27th aspects, in which the display mode of a specific image is changed depending on the viewpoint used to generate the virtual viewpoint image.

[0034] A 29th aspect of the technology of the present disclosure is an image processing device comprising a processor and a memory connected to or built into the processor, wherein the processor acquires a virtual viewpoint image generated based on a plurality of captured images, and outputs data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with a plurality of regions related to the virtual viewpoint image and specific image-related information related to a specific image not included in the plurality of captured images.

[0035] A thirtieth aspect of the technique of the present disclosure is the image processing device according to the twenty-ninth aspect, in which the timings at which the multiple regions are displayed are different.

[0036] A 31st aspect of the technology of the present disclosure is an image processing method that includes obtaining a virtual viewpoint image generated based on a plurality of captured images, and outputting first data for displaying a first specific image selected from a plurality of specific images in a first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images that are not included in the plurality of captured images.

[0037] A 32nd aspect of the technology of the present disclosure is an image processing method that includes obtaining a virtual viewpoint image generated based on a plurality of captured images, and outputting data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with a plurality of regions related to the virtual viewpoint image and specific image-related information related to a specific image not included in the plurality of captured images.

[0038] A 33rd aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a virtual viewpoint image generated based on a plurality of captured images, and outputting first data for displaying a first specific image selected from a plurality of specific images in a first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images not included in the plurality of captured images.

[0039] A 34th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a virtual viewpoint image generated based on a plurality of captured images, and outputting data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with a plurality of regions related to the virtual viewpoint image and specific image-related information related to a specific image not included in the plurality of captured images. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of an image processing system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the hardware configuration of an electrical system of a user device. [Figure 3] FIG. 2 is a block diagram showing an example of main functions of a CPU of the image processing device. [Figure 4] 10A and 10B are conceptual diagrams illustrating an example of processing content of a reception screen generating unit and an example of display content on a display of a user device. [Figure 5]10 is a screen diagram showing an example of a display mode of a reception screen when the operation mode of the user device is a viewpoint setting mode. FIG. [Figure 6] 10 is a screen diagram showing an example of a display mode of a reception screen when the operation mode of the user device is a gaze point setting mode. FIG. [Figure 7] 10 is a block diagram showing an example of the contents of viewpoint information and an example of a manner in which viewpoint information is transmitted from a user device to an image processing device. FIG. [Figure 8] FIG. 10 is a conceptual diagram illustrating an example of processing content of a virtual viewpoint image generating unit. [Figure 9] FIG. 3 is a conceptual diagram showing an example of processing content of a screen data generating unit according to the first embodiment. [Figure 10] FIG. 3 is a conceptual diagram showing an example of processing content of a screen data generating unit according to the first embodiment. [Figure 11] 6 is a flowchart showing an example of the contents stored in a storage included in the image processing device of the image processing system according to the first embodiment. [Figure 12] FIG. 3 is a conceptual diagram showing an example of processing content of a screen data generating unit according to the first embodiment. [Figure 13] 3 is a schematic screen diagram showing an example of a virtual viewpoint video screen displayed on a touch panel display of a user device according to the first embodiment. FIG. [Figure 14] 10 is a flowchart showing an example of the flow of a screen generation process according to the first embodiment. [Figure 15] FIG. 11 is a conceptual diagram showing an example of processing content of a screen data generating unit according to the second embodiment. [Figure 16] FIG. 10 is a conceptual diagram showing an example of the contents stored in a storage included in an image processing device of an image processing system according to a second embodiment. [Figure 17] FIG. 11 is a conceptual diagram showing an example of processing content of a screen data generating unit according to the second embodiment. [Figure 18] FIG. 11 is a schematic screen diagram showing an example of a virtual viewpoint video screen displayed on a touch panel display of a user device according to the second embodiment. [Figure 19A]10 is a flowchart showing an example of the flow of a screen generation process according to the second embodiment. [Figure 19B] This is a continuation of the flowchart shown in FIG. 19A. [Figure 20] FIG. 10 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a first modified example. [Figure 21] FIG. 10 is a conceptual diagram showing an example of advertising video related information according to a first modified example. [Figure 22] FIG. 10 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a second modified example. [Figure 23] FIG. 11 is a conceptual diagram showing an example of advertising video related information according to a second modified example. [Figure 24] FIG. 13 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a third modified example. [Figure 25] FIG. 13 is a conceptual diagram showing an example of advertising video related information according to a third modified example. [Figure 26] FIG. 13 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a fourth modified example. [Figure 27] FIG. 13 is a conceptual diagram showing an example of advertising video related information according to a fourth modified example. [Figure 28] FIG. 13 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a fifth modified example. [Figure 29] FIG. 13 is a conceptual diagram showing an example of processing contents of a screen data generating unit according to a fifth modified example. [Figure 30] FIG. 13 is a conceptual diagram showing an example of advertising video related information according to a sixth modified example. [Figure 31] FIG. 13 is a conceptual diagram showing an example of advertising video related information according to a sixth modified example. [Figure 32] FIG. 13 is a conceptual diagram showing an example of first advertising area related information and second advertising area related information according to a seventh modified example. [Figure 33] FIG. 13 is a conceptual diagram showing an example of advertising video related information according to a seventh modified example. [Figure 34] FIG. 13 is a conceptual diagram showing an example of processing contents of a screen data generating unit according to a ninth modified example. [Figure 35] FIG. 23 is a conceptual diagram showing an example of processing contents of a screen data generating unit according to a tenth modified example. [Figure 36] FIG. 10 is a conceptual diagram showing an example of how a screen generation processing program stored in a storage medium is installed in a computer of an image processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0041] An example of an embodiment of an image processing device, an image processing method, and a program according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0042] First, the terms used in the following description will be explained.

[0043] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". LAN is an abbreviation for "Local Area Network." USB is an abbreviation for "Universal Serial Bus." HMD is an abbreviation for "Head Mounted Display." LTE is an abbreviation for "Long Term Evolution." 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)." TDM is an abbreviation for "Time-Division Multiplexing." AI is an abbreviation for "Artificial Intelligence." In this specification, a subject included in an image (in the sense of an image including still images and moving images) refers to a subject included in the image as an image (for example, an electronic image).In addition, in the explanations of this specification, "match" refers not only to a perfect match, but also to a match that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the intent of the technology of the present disclosure.

[0044] [First embodiment] As an example, as shown in FIG. 1, an image processing system 2 includes an image processing device 10 and a user device 12.

[0045] In the first embodiment, a server is applied as an example of the image processing device 10. The server is realized by, for example, a mainframe. However, this is merely an example, and the server may be realized by network computing such as cloud computing, fog computing, edge computing, or grid computing. Furthermore, the image processing device 10 may be a plurality of servers, a workstation, a personal computer, a device combining at least one workstation and at least one personal computer, or a device combining at least one workstation, at least one personal computer, and at least one server.

[0046] In the first embodiment, a smartphone is used as an example of the user device 12. However, the smartphone is merely an example, and the user device 12 may be, for example, a personal computer, a tablet terminal, an HMD, or another portable multi-function terminal.

[0047] In the first embodiment, the image processing device 10 and the user device 12 are communicatively connected via, for example, a base station (not shown). The communication standards used in the base station include wireless communication standards such as the 5G standard and / or the LTE standard, wireless communication standards such as the WiFi (802.11) standard and / or the Bluetooth (registered trademark) standard, and wired communication standards such as the TDM standard and / or the Ethernet (registered trademark) standard.

[0048] The image processing device 10 acquires an image and transmits the acquired image to the user device 12. Here, the image refers to, for example, a captured image 64 (see FIG. 4) obtained by capturing an image, an image generated based on the captured image 64 (see FIG. 4, etc.), etc. An example of an image generated based on the captured image (see FIG. 4) is a virtual viewpoint image 76 (see FIG. 8, etc.).

[0049] The user device 12 is used by a user 14. The user device 12 includes a touch panel display 16. The touch panel display 16 is realized by a display 18 and a touch panel 20. An example of the display 18 is an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0050] The touch panel display 16 is formed by overlaying the touch panel 20 on the display area of ​​the display 18, or by using an in-cell type in which the touch panel function is built into the display 18. Note that the in-cell type is merely an example, and the display may also be an out-cell type or an on-cell type.

[0051] The user device 12 executes processing in accordance with instructions received from a user via the touch panel 20 or the like. For example, the user device 12 exchanges various types of information with the image processing device 10 in accordance with instructions received from a user via the touch panel 20 or the like.

[0052] The user device 12 receives the image transmitted from the image processing device 10 and displays the received image on the display 18. The user 14 views the image displayed on the display 18.

[0053] The image processing device 10 includes a computer 22, a transmission / reception device 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure, and includes a processor 28, a storage 30, and a RAM 32. The image processing device 10 includes a bus 34, and the processor 28, the storage 30, and the RAM 32 are connected via the bus 34. In the example shown in FIG. 1, for convenience of illustration, a single bus is shown as the bus 34, but multiple buses may be used. The bus 34 may also include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, etc.

[0054] The processor 28 is an example of a "processor" according to the technology of the present disclosure. The processor 28 controls the entire image processing device 10. For example, the processor 28 has a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing image processing.

[0055] The storage 30 stores various parameters, various programs, etc. Examples of the storage 30 include an EEPROM, an SSD, and / or an HDD. The storage 30 is an example of a "memory" according to the technology of the present disclosure. Various information is temporarily stored in the RAM 32. The RAM 32 is used as a work memory by the processor 28.

[0056] The transceiver 24 is connected to the bus 34. The transceiver 24 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the user device 12 via a base station (not shown) under the control of the processor 28. In other words, the processor 28 transmits and receives various types of information to and from the user device 12 via the transceiver 24.

[0057] The communication I / F 26 is realized by, for example, a device having an FPGA. The communication I / F 26 is connected to a plurality of image capture devices 36 via LAN cables (not shown). The image capture devices 36 are image capture devices having CMOS image sensors and equipped with optical zoom functions and / or digital zoom functions. Note that other types of image sensors, such as CCD image sensors, may be used instead of the CMOS image sensors.

[0058] For example, multiple imaging devices 36 are installed in a soccer stadium (not shown) and capture images of subjects within the soccer stadium. Captured images 64 (see FIG. 4) obtained by capturing images of subjects by the imaging devices 36 are used to generate, for example, a virtual viewpoint image 76 (see FIG. 8, etc.). For this reason, each of the multiple imaging devices 36 is installed in a different location within the soccer stadium, i.e., a location where multiple captured images 64 (see FIG. 4) from which the virtual viewpoint image 76 (see FIG. 8, etc.) can be generated are obtained. Here, the multiple captured images 64 are an example of the "multiple captured images" according to the technology of the present disclosure.

[0059] The soccer stadium is a three-dimensional area including a soccer field and spectator stands constructed to surround the soccer field, and is the subject of observation by the user 14. The observer, i.e., the user 14, can observe the inside of the soccer stadium from the spectator stands or a location outside the soccer stadium through an image displayed by the display 18 of the user device 12.

[0060] Here, a soccer stadium is used as an example of a location where multiple imaging devices 36 are installed, but the technology of the present disclosure is not limited to this, and the location where multiple imaging devices 36 are installed may be any location where multiple imaging devices 36 can be installed, such as a baseball stadium, rugby field, curling court, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater venue.

[0061] The communication I / F 26 is connected to the bus 34 and controls the exchange of various information between the processor 28 and the multiple imaging devices 36. For example, the communication I / F 26 controls the multiple imaging devices 36 in accordance with a request from the processor 28. The communication I / F 26 outputs a captured image 64 (see FIG. 4) obtained by capturing an image by each of the multiple imaging devices 36 to the processor 28. Note that although the communication I / F 26 is exemplified here as a wired communication I / F, it may also be a wireless communication I / F such as a high-speed wireless LAN.

[0062] The storage 30 stores a screen generation processing program 38. The screen generation processing program 38 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the screen generation processing program 38 from the storage 30 and executes the screen generation processing program 38 on the RAM 32 to perform screen generation processing (see FIG. 14).

[0063] 2, the user device 12 includes a display 18, a computer 40, an imaging device 42, a transceiver device 44, a speaker 46, a microphone 48, and an accepting device 50. The computer 40 includes a processor 52, storage 54, and RAM 56. The user device 12 includes a bus 58, and the processor 52, storage 54, and RAM 56 are connected via the bus 58.

[0064] 2, for convenience of illustration, one bus is shown as bus 58, but multiple buses may be used. Bus 58 may also include a serial bus or a parallel bus configured from a data bus, an address bus, a control bus, and the like.

[0065] The processor 52 controls the entire user device 12. The processor 52 includes, for example, a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing image processing.

[0066] The storage 54 stores various parameters, various programs, etc. An example of the storage 54 is an EEPROM. Various information is temporarily stored in the RAM 56. The RAM 56 is used as a work memory by the processor 52. The processor 52 reads various programs from the storage 54 and executes the programs on the RAM 56 to perform processing in accordance with the programs.

[0067] The imaging device 42 is an imaging device having a CMOS image sensor and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The imaging device 42 is connected to a bus 58, and the processor 52 controls the imaging device 42. The captured image obtained by imaging the imaging device 42 is acquired by the processor 52 via the bus 58.

[0068] The transmitting / receiving device 44 is connected to the bus 58. The transmitting / receiving device 44 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the image processing device 10 via a base station (not shown) under the control of the processor 52. That is, the processor 52 transmits and receives various types of information to and from the image processing device 10 via the transmitting / receiving device 44.

[0069] The speaker 46 converts the electrical signal into sound. The speaker 46 is connected to the bus 58. The speaker 46 receives the electrical signal output from the processor 52 via the bus 58, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the user device 12.

[0070] The microphone 48 converts the collected sound into an electrical signal. The microphone 48 is connected to a bus 58. The electrical signal obtained by converting the sound collected by the microphone 48 is acquired by the processor 52 via the bus 58.

[0071] The accepting device 50 accepts instructions from the user 14 or the like. Examples of the accepting device 50 include the touch panel 20 and hard keys (not shown). The accepting device 50 is connected to a bus 58, and the instructions accepted by the accepting device 50 are acquired by a processor 52.

[0072] As an example, as shown in FIG. 3, in the image processing device 10, the processor 28 reads a screen generation processing program 38 from the storage 30 and executes the screen generation processing program 38 on the RAM 32, thereby operating as a reception screen generation unit 28A, a viewpoint information acquisition unit 28B, a virtual viewpoint image generation unit 28C, and a screen data generation unit 28D.

[0073] The storage 30 stores a plurality of advertising videos 60, which are used by the screen data generating unit 28D and the like. The advertising videos 60 are an example of images not included in the plurality of captured images 64 obtained by capturing images using the plurality of imaging devices 36. The advertising videos 60 are also an example of images created in a process different from the virtual viewpoint video 78 (see FIGS. 4 and 8 to 10). The advertising videos 60 are also an example of images showing advertisements (e.g., moving images obtained by capturing an image of a subject in a real space region with a camera, virtual viewpoint video created in a process different from the virtual viewpoint video 78, and / or animation). The video showing an advertisement refers to, for example, a moving image in which a plurality of frames of images created as advertising images are arranged in chronological order. An example of the advertising videos 60 is a video provided by a sponsor or the like supporting the construction of a system (e.g., the image processing system 2 shown in FIG. 1) that allows users 14 to view various videos including the virtual viewpoint video 78 (see FIGS. 4 and 8 to 10).

[0074] Here, a moving image is used as an example of the advertising video 60, but the technology of the present disclosure is not limited to this, and the advertising video 60 may be a single-frame advertising image or an image used for purposes other than advertising. The advertising video 60 is merely an example, and may be other types of moving image or still image. The advertising video 60 is an example of a "specific image" and a "first specific image" according to the technology of the present disclosure.

[0075] 4, a reception screen 66 and a virtual viewpoint video screen 68 are displayed on the touch panel display 16 of the user device 12. In the example shown in Fig. 4, the reception screen 66 and the virtual viewpoint video screen 68 are displayed side by side on the touch panel display 16. Note that this is merely an example, and the reception screen 66 and the virtual viewpoint video screen 68 may be switched and displayed in accordance with an instruction given to the touch panel display 16 by the user 14, or the reception screen 66 and the virtual viewpoint video screen 68 may be displayed separately on different display devices.

[0076] Also, in the example shown in Figure 4, the reception screen 66 is displayed on the touch panel display 16 of the user device 12, but the technology of the present disclosure is not limited to this, and for example, the reception screen 66 may be displayed on a display connected to a device (e.g., a workstation and / or a personal computer) used by a person who creates or edits the virtual viewpoint image 78 (see Figure 8).

[0077] The user device 12 communicates with the image processing device 10 to acquire a virtual viewpoint video 78 (see also FIG. 8 ) from the image processing device 10. The virtual viewpoint video 78 (see also FIG. 8 ) acquired by the user device 12 from the image processing device 10 is displayed on the virtual viewpoint video screen 68 of the touch panel display 16. The virtual viewpoint video 78 is an example of a "virtual viewpoint image" according to the technology of the present disclosure. Note that, although the virtual viewpoint video 78 is exemplified as a moving image in the first embodiment, the technology of the present disclosure also applies to a virtual viewpoint image 76 that is a still image.

[0078] The virtual viewpoint video screen 68 has a first advertising area 79. That is, the first advertising area 79 is displayed in the virtual viewpoint video 78. The first advertising area 79 is an example of a "first area" according to the technology of the present disclosure.

[0079] In the example shown in Fig. 4, the first advertising area 79 is rectangular and is displayed in the upper right corner when viewed from the front of the virtual viewpoint video screen 68. However, the shape, size, and display position of the first advertising area 79 shown in Fig. 4 are merely an example, and the shape, size, and display position may be different from those shown in the example of Fig. 4. As will be described in more detail later, the first advertising area 79 displays an advertising video 60 (see Figs. 3 and 11 to 13).

[0080] The first advertising area 79 is an area related to the virtual viewpoint video 78. Here, the concept of an area related to the virtual viewpoint video 78 includes concepts such as an area displayed in the virtual viewpoint video 78, an area where an image related to the virtual viewpoint video 78 (for example, the overhead view video 72) is displayed, an area displayed until the virtual viewpoint video 78 is displayed, and an area displayed at the display timing of the virtual viewpoint video 78 (for example, a timing determined according to the content of the virtual viewpoint video 78, a timing determined based on the timing when the display of the virtual viewpoint video 78 starts, or a timing determined based on the timing when the display of the virtual viewpoint video 78 ends).

[0081] The first advertising area 79 is displayed superimposed on the virtual viewpoint video 78. In the first embodiment, the first advertising area 79 is simply displayed superimposed on the virtual viewpoint video 78. However, this is merely an example. One method for displaying the first advertising area 79 superimposed on the virtual viewpoint video 78 is, for example, alpha blending. In this case, the alpha value may be changed. The magnitude of the alpha value and / or the timing of changing the alpha value may be determined according to various information related to the virtual viewpoint video 78 (for example, the content of the virtual viewpoint video 78, the timing when the display of the virtual viewpoint video 78 starts, and / or the timing when the display of the virtual viewpoint video 78 ends), etc.

[0082] Here, an example is given in which the first advertising area 79 is displayed superimposed on the virtual viewpoint video 78, but this is merely an example, and the first advertising area 79 may also be displayed embedded in the virtual viewpoint video 78.

[0083] The user device 12 communicates with the image processing device 10 to acquire reception screen data 70 representing a reception screen 66 from the image processing device 10. The touch panel display 16 displays the reception screen 66 represented by the reception screen data 70 acquired by the user device 12 from the image processing device 10.

[0084] The reception screen 66 includes an overhead view video screen 66A, a guidance message display area 66B, a decision key 66C, and a cancel key 66D, and displays various information necessary for generating the virtual viewpoint video 78. The user 14 refers to the reception screen 66 and gives instructions to the user device 12. The instructions from the user 14 are received by, for example, the touch panel display 16.

[0085] The overhead video screen 66A displays an overhead video 72. The overhead video 72 is a moving image showing the state of the soccer stadium when observed from an overhead view, and is generated based on a plurality of captured images 64 obtained by capturing images by at least one of the plurality of imaging devices 36. Examples of the overhead video 72 include recorded video and / or live broadcast video.

[0086] The guidance message display area 66B displays various messages indicating the contents of operations required of the user 14. The operations required of the user 14 refer to, for example, operations required to generate the virtual viewpoint video 78 (for example, an operation to set a viewpoint, an operation to set a point of interest, etc.).

[0087] The display content of the guidance message display area 66B changes depending on the operation mode of the user device 12. For example, the user device 12 has, as operation modes, a viewpoint setting mode for setting a viewpoint and a gaze point setting mode for setting a gaze point, and the display content of the guidance message display area 66B differs between the viewpoint setting mode and the gaze point setting mode.

[0088] The enter key 66C and the cancel key 66D are both soft keys. The enter key 66C is pressed by the user 14 when deciding on an instruction accepted on the reception screen 66. The cancel key 66D is pressed by the user 14 when canceling an instruction accepted on the reception screen 66.

[0089] The reception screen generation unit 28A acquires a plurality of captured images 64 from a plurality of imaging devices 36. The captured images 64 include imaging condition information 64A. The imaging condition information 64A refers to information indicating the imaging conditions. Examples of the imaging conditions include three-dimensional coordinates that can identify the installation position of the imaging device 36, the imaging direction of the imaging device 36, the angle of view used in imaging by the imaging device 36, and the zoom magnification applied to the imaging device 36.

[0090] The reception screen generation unit 28A generates an overhead video 72 based on a plurality of captured images 64 acquired from a plurality of imaging devices 36. Then, the reception screen generation unit 28A generates data showing a reception screen 66 including the overhead video 72 as reception screen data 70.

[0091] The reception screen generation unit 28A outputs the reception screen data 70 to the transmission / reception device 24. The transmission / reception device 24 transmits the reception screen data 70 input from the reception screen generation unit 28A to the user device 12. The user device 12 receives the reception screen data 70 transmitted from the transmission / reception device 24 at the transmission / reception device 44 (see FIG. 3 ). The reception screen 66 indicated by the reception screen data 70 received by the transmission / reception device 44 is displayed on the touch panel display 16.

[0092] As an example, as shown in FIG. 5, when the operation mode of the user device 12 is the viewpoint setting mode, a message 66B1 is displayed in the guidance message display area 66B of the reception screen 66. The message 66B1 is a message that prompts the user 14 to specify the viewpoint to be used in generating the virtual viewpoint video 78 (see FIGS. 4 and 8). Here, the viewpoint refers to a virtual viewpoint from which the inside of the soccer stadium is observed. The virtual viewpoint refers to the position where a virtual camera that captures an image of a subject is installed, rather than the position where an actual camera such as a physical camera (e.g., the imaging device 36) that captures an image of the subject is installed.

[0093] Touch panel display 16 accepts an instruction from user 14 while message 66B1 is displayed in guidance message display area 66B. In this case, the instruction from user 14 refers to an instruction of a viewpoint. The viewpoint corresponds to the position of a pixel in overhead view image 72. The position of a pixel in overhead view image 72 corresponds to a position in the soccer stadium. The instruction of the viewpoint is performed by user 14 indicating the position of a pixel in overhead view image 72 via touch panel display 16. Note that the viewpoint may be three-dimensional coordinates corresponding to a three-dimensional position in overhead view image 72. Any method can be used to indicate the three-dimensional position. For example, user 14 may directly input the three-dimensional coordinate position, or the three-dimensional coordinate position may be specified by displaying two images of the soccer stadium viewed from two planes perpendicular to each other and specifying the pixel position in each image.

[0094] In the example shown in FIG. 5, a viewpoint path P1, which is a path for observing a subject, is shown as an example of a viewpoint. The viewpoint path P1 is a collection of multiple viewpoints arranged in a line from a start point P1s to an end point P1e. ​​The viewpoint path P1 is defined along a path (a meandering path from the start point P1s to the end point P1e in the example shown in FIG. 5) along which the user 14 slides (swipes) their fingertip 14A over an area of ​​the entire area of ​​the touch panel 20 corresponding to the display area of ​​the overhead image 72. The observation time from the viewpoint path P1 (e.g., the time spent observing between two different viewpoints and / or the time spent stationary at a single point) is defined by the speed of the slide made on the touch panel display 16 when forming the viewpoint path P1 via the touch panel display 16, the time spent staying at one viewpoint on the viewpoint path P1 (e.g., the time spent long-pressing), and the like.

[0095] In the example shown in FIG. 5, when the viewpoint path P1 is confirmed, the enter key 66C is turned on, and when the viewpoint path P1 is cancelled, the cancel key 66D is turned on.

[0096] 5, only viewpoint path P1 is set, but this is merely an example, and multiple viewpoint paths may be set. Also, instead of being limited to viewpoint paths, multiple discontinuous viewpoints or a single viewpoint may be set.

[0097] As an example, as shown in FIG. 6, when the operation mode of the user device 12 is the gaze point setting mode, a message 66B2 is displayed in the guidance message display area 66B of the reception screen 66. The message 66B2 is a message that prompts the user 14 to specify a gaze point to be used in generating a virtual viewpoint image 78 (see FIG. 8). Here, the gaze point refers to a point that is virtually gazed upon when observing the inside of a soccer stadium from a viewpoint. Once the viewpoint and gaze point are set, the virtual gaze direction (imaging direction of the virtual camera) is also uniquely determined. The virtual gaze direction refers to the direction from the viewpoint toward the gaze point.

[0098] Touch panel display 16 accepts an instruction from user 14 while message 66B2 is displayed in guidance message display area 66B. In this case, the instruction from user 14 refers to an instruction for a point of gaze. The point of gaze corresponds to the position of a pixel in overhead view image 72. The position of the pixel in overhead view image 72 corresponds to a position in the soccer stadium. The instruction for the point of gaze is made by user 14 indicating the position of a pixel in overhead view image 72 via touch panel display 16. In the example shown in FIG. 6, point of gaze GP is shown. Point of gaze GP is defined according to a location where user 14 touches with fingertip 14A in an area of ​​touch panel display 16 that corresponds to the display area of ​​overhead view image 72. In the example shown in FIG. 6, to confirm point of gaze GP, enter key 66C is pressed, and to cancel point of gaze GP, cancel key 66D is pressed. Note that the point of gaze may be a three-dimensional coordinate corresponding to a three-dimensional position in overhead view image 72. As with the method for specifying the viewpoint position, any method can be used to specify the three-dimensional position.

[0099] 6, only the gaze point GP is specified, but this is merely an example, and there may be multiple gaze points, or a path (gazing point path) in which multiple gaze points are arranged in a line. The gaze point path may be one or multiple.

[0100] 7, the processor 52 of the user device 12 generates viewpoint information 74 based on the viewpoint path P1 and the gaze point GP. The viewpoint information 74 is an example of the "first viewpoint information" and "second viewpoint information" according to the technology of the present disclosure.

[0101] The viewpoint information 74 is information used to generate a virtual viewpoint video 78 (see FIG. 8 ). The viewpoint information 74 includes total time information 74A, viewpoint path information 74B, required time information 74C, elapsed time information 74D, movement speed information 74E, angle of view information 74F, and point of interest information 74G.

[0102] The total time information 74A is information indicating the total time (hereinafter simply referred to as "total time" or "display time") that a virtual viewpoint video 78 (see FIG. 8) generated based on one or more viewpoint paths (for example, only viewpoint path P1, or multiple viewpoint paths including viewpoint path P1) determined in the viewpoint setting mode is played back at a standard playback speed. The total time corresponds to the time that fingertip 14A is slid across touch panel display 16 to create multiple viewpoint paths.

[0103] The viewpoint path information 74B is information indicating the viewpoint path P1 (see FIG. 5) confirmed in the viewpoint setting mode (for example, coordinates specifying the position of pixels of the viewpoint path P1 in the overhead image 72). The viewpoint path information 74B is information capable of specifying the position of the viewpoint (hereinafter also referred to as the "viewpoint position"). The viewpoint position refers to, for example, the position of the virtual camera described above. Here, as an example of the viewpoint position, the position of pixels in the overhead image 72 of one viewpoint included in the viewpoint path P1 confirmed in the viewpoint setting mode is applied. An example of information specifying the position of pixels of the viewpoint path P1 in the overhead image 72 is coordinates specifying the position of pixels of the viewpoint path P1 in the overhead image 72.

[0104] The viewpoint path P1 includes a start point P1s and an end point P1e (see FIG. 5). Therefore, the multiple viewpoint position information indicating all viewpoint positions included in the viewpoint path P1 also includes start point position information (hereinafter also simply referred to as "start point position information") that can identify the position of the start point P1s, and end point position information (hereinafter also simply referred to as "end point position information") that can identify the position of the end point P1e. ​​An example of the start point position information is coordinates that can identify the position of the pixel of the start point P1s in the overhead image 72. An example of the end point position information is coordinates that can identify the position of the pixel of the end point P1e in the overhead image 72.

[0105] The required time information 74C is information indicating the required time (hereinafter also simply referred to as "required time") required for the viewpoint for observing the subject on the viewpoint path P1 to move from a first position to a second position different from the first position. Here, the first position refers to the start point P1s (see FIGS. 5 and 6), and the second position refers to, for example, the end point P1e (see FIGS. 5 and 6). Note that this is merely an example, and the first position may be the start point P1s (see FIGS. 5 and 6) and the second position may be a viewpoint position at an intermediate point on the viewpoint path P1, or the first position may be a viewpoint position at an intermediate point on the viewpoint path P1 and the second position may be the end point P1e (see FIGS. 5 and 6).

[0106] The elapsed time information 74D is information indicating the position of the viewpoint from which the subject is observed on the viewpoint path P1 and the elapsed time corresponding to the viewpoint position. The elapsed time corresponding to the viewpoint position (hereinafter simply referred to as "elapsed time") refers to, for example, the time during which the viewpoint remains stationary at a certain viewpoint position on the viewpoint path P1.

[0107] The movement speed information 74E is information that can identify the movement speed of the position of the viewpoint from which the subject is observed on the viewpoint path P1, i.e., the speed at which the viewpoint moves on the viewpoint path P1. The movement speed of the viewpoint position (hereinafter also simply referred to as "movement speed") refers to, for example, the speed of sliding performed on the touch panel display 16 when forming the viewpoint path P1 via the touch panel display 16. The movement speed information 74E is associated with each viewpoint in the viewpoint path P1.

[0108] The angle of view information 74F is information indicating the angle of view (hereinafter also simply referred to as "angle of view"). Here, the angle of view refers to the viewing angle at which the subject is observed on the viewpoint path P1. In the first embodiment, the angle of view is fixed to a predetermined angle (for example, 100 degrees). However, this is merely an example, and the angle of view may also be determined according to the movement speed.

[0109] When the angle of view is determined according to the moving speed, for example, within a range in which the upper limit (e.g., 150 degrees) and the lower limit (e.g., 15 degrees) of the angle of view are determined, the angle of view becomes narrower as the moving speed decreases. Alternatively, the angle of view may become narrower as the moving speed increases.

[0110] The angle of view may also be determined according to the elapsed time. When the angle of view is determined according to the elapsed time, for example, the angle of view may be minimized when the elapsed time exceeds a first predetermined time (e.g., 3 seconds), or maximized when the elapsed time exceeds the first predetermined time.

[0111] The angle of view may also be determined according to an instruction received by the receiving device 50. In this case, the receiving device 50 may receive an instruction regarding the viewpoint position at which the angle of view is changed on the viewpoint path P1 and the angle of view after the change.

[0112] The point of interest information 74G is information that can identify the position of the point of interest GP that has been determined in the point of interest setting mode (for example, coordinates that identify the pixel position of the point of interest GP in the overhead view image 72).

[0113] The processor 52 outputs the viewpoint information 74 to the transmission / reception device 44. The transmission / reception device 44 transmits the viewpoint information 74 input from the processor 52 to the image processing device 10. The transmission / reception device 24 of the image processing device 10 receives the viewpoint information 74. The viewpoint information acquisition unit 28B of the image processing device 10 acquires the viewpoint information 74 received by the transmission / reception device 24.

[0114] The processor 52 outputs the viewpoint information 74 to the transmission / reception device 44. The transmission / reception device 44 transmits the viewpoint information 74 input from the processor 52 to the image processing device 10. The transmission / reception device 24 of the image processing device 10 receives the viewpoint information 74 transmitted from the image processing device 10. The viewpoint information acquisition unit 28B of the image processing device 10 acquires the viewpoint information 74 received by the transmission / reception device 24.

[0115] As an example, as shown in FIG. 8 , the virtual viewpoint image generation unit 28C generates, based on a plurality of captured images 64 and viewpoint information 74, a virtual viewpoint image 76, which is an image showing the appearance of a subject when the subject is observed from a viewpoint specified by the viewpoint information 74. For example, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from a plurality of imaging devices 36 in accordance with the viewpoint information 74, and generates a virtual viewpoint image 76 for each viewpoint on a viewpoint path P1 based on the acquired plurality of captured images 64. That is, the virtual viewpoint image generation unit 28C generates a plurality of frames of virtual viewpoint images 76 in accordance with the viewpoint path P1. The plurality of frames of virtual viewpoint images 76 generated in accordance with the viewpoint path P1 refer to the virtual viewpoint images 76 generated for each viewpoint on the viewpoint path P1. The virtual viewpoint image generation unit 28C generates a virtual viewpoint video 78 by chronologically arranging the plurality of frames of virtual viewpoint images 76. In addition, even if there is a viewpoint path other than viewpoint path P1, the virtual viewpoint image generation unit 28C generates multiple frames of virtual viewpoint images 76, as in the case of viewpoint path P1, and generates a virtual viewpoint video 78 by arranging the generated multiple frames of virtual viewpoint images 76 in chronological order.

[0116] The virtual viewpoint video 78 is a moving image in which a plurality of frames of virtual viewpoint images 76 are arranged in time series. The person who views the virtual viewpoint video 78 is, for example, the user 14. The virtual viewpoint video 78 is viewed by the user 14 via the display 18 of the user device 12. For example, the plurality of frames of the virtual viewpoint image 76 are displayed from the first frame to the last frame on the virtual viewpoint video screen 68 (see FIG. 4 ) of the display 18 of the user device 12 at a predetermined frame rate (for example, several tens of frames per second), and are viewed by the user 14 as the virtual viewpoint video 78.

[0117] As an example, as shown in Fig. 9, the screen data generation unit 28D generates virtual viewpoint video screen data 80 based on the virtual viewpoint video 78 generated by the virtual viewpoint image generation unit 28C. As will be described in detail later, the screen data generation unit 28D selects an advertising video 60 from a plurality of advertising videos 60 (see Figs. 3 and 11). The virtual viewpoint video screen data 80 is data (e.g., data indicating a virtual viewpoint video screen 68) for displaying the advertising video 60 selected by the screen data generation unit 28D from the plurality of advertising videos 60 (see Figs. 3 and 11) in a first advertising area 79. In the first embodiment, the virtual viewpoint video screen data 80 is an example of "first data" according to the technology of the present disclosure.

[0118] As an example, as shown in FIG. 10, the screen data generation unit 28D generates first advertising area-related information 82 based on various information. The first advertising area-related information 82 is information associated with the first advertising area 79. Examples of the various information used to generate the first advertising area-related information 82 include the virtual viewpoint video 78 generated by the virtual viewpoint image generation unit 28C, the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B (for example, the viewpoint information 74 used to generate the virtual viewpoint video 78 by the virtual viewpoint image generation unit 28C shown in FIG. 8), and / or information corresponding to an instruction accepted by the acceptance device 50 or the like. The first advertising area-related information 82 is an example of the "first information" and "area-related information" according to the technology of the present disclosure.

[0119] The first advertising area related information 82 includes first content related information 82 A. The first content related information 82 A is information related to the content of the virtual viewpoint video 78.

[0120] A first example of the first content-related information 82A is the title of the virtual viewpoint video 78. The title of the virtual viewpoint video 78 may be, for example, a title determined according to an instruction received by the reception device 50 or the like, or may be a title determined based on the virtual viewpoint video 78. The title determined based on the virtual viewpoint video 78 is generated, for example, by performing an AI-based object recognition process. In this case, for example, the screen data generation unit 28D performs an AI-based object recognition process on the virtual viewpoint video 78 to identify the type of object included in the virtual viewpoint video 78. The screen data generation unit 28D then derives a title appropriate for the type of object included in the virtual viewpoint video 78. The title appropriate for the type of object is derived, for example, from a title derivation table (not shown) that receives the type of object as an input and outputs a title. The title derivation table may be a table that receives not only the type of object but also information such as the viewpoint information 74 as an input and outputs a title.

[0121] A second example of the first content-related information 82A is the name of a sport that the user 14 is watching through the virtual perspective video 78 and / or the overhead-view video 72. The sport name may be determined according to instructions received by the reception device 50 or the like, or may be determined based on the virtual perspective video 78. The sport name determined based on the virtual perspective video 78 is generated, for example, by performing an AI-based object recognition process. In this case, for example, the screen data generation unit 28D performs an AI-based object recognition process on the virtual perspective video 78 to identify the type of object included in the virtual perspective video 78. The screen data generation unit 28D then derives a sport name appropriate for the type of object included in the virtual perspective video 78. The sport name appropriate for the type of object is derived, for example, from a sport name derivation table (not shown) that receives the type of object as input and outputs the name of the sport.

[0122] A third example of the first content-related information 82A is the name of the main subject that the user 14 sees through the virtual viewpoint video 78 and / or the overhead-view video 72 (e.g., the subject that appears most frequently in the virtual viewpoint video 78, the subject that appears for the longest time in the virtual viewpoint video 78, and / or the subject that appears in the virtual viewpoint video 78 at a size larger than a predetermined size for at least a predetermined number of frames, etc.). The name of the main subject may be, for example, a name determined according to an instruction received by the reception device 50 or the like, or may be a name determined based on the virtual viewpoint video 78. The name determined based on the virtual viewpoint video 78 is generated, for example, by performing an AI-based subject recognition process. In this case, for example, the screen data generation unit 28D performs an AI-based subject recognition process on the virtual viewpoint video 78 to identify the type of subject included in the virtual viewpoint video 78. Then, the screen data generation unit 28D derives a name appropriate for the type of subject included in the virtual viewpoint video 78. A name suitable for the type of subject is derived from, for example, a name derivation table (not shown) that takes the type of subject as input and the name of the subject as output.

[0123] Although an AI-based subject recognition process is exemplified here, this is merely one example, and other subject recognition processes such as a template matching-based subject recognition process may also be used.

[0124] In this way, when the type of subject is identified by performing the subject recognition process on the virtual viewpoint video 78 and the first content-related information 82A is determined in accordance with the identified type of subject, the type of subject identified by the screen data generation unit 28D is changed in accordance with the viewpoint information 74 required to generate the virtual viewpoint video 78, and therefore the first content-related information 82A is also changed accordingly. For example, the first content-related information 82A is changed by the screen data generation unit 28D in accordance with at least one of the total time information 74A, viewpoint path information 74B, required time information 74C, elapsed time information 74D, moving speed information 74E, angle of view information 74F, and point of interest information 74G included in the viewpoint information 74.

[0125] The screen data generation unit 28D associates the generated first advertising area related information 82 with the first advertising area 79. Note that information associating the first advertising area 79 with the first advertising area related information 82 may be stored in a storage device such as the storage 30 together with the virtual viewpoint video 78 or separately from the virtual viewpoint video 78.

[0126] As an example, as shown in FIG. 11 , a plurality of pieces of advertising video related information 84 are stored in the storage 30. The advertising video related information 84 is information related to the advertising video 60, and is individually associated with each of the plurality of advertising videos 60. In the first embodiment, for example, the content of the advertising video related information 84 differs between the pieces of advertising video related information 84 associated with each advertising video 60. Note that the plurality of pieces of advertising video related information 84 are an example of "specific image related information" according to the technology of the present disclosure.

[0127] The advertising video related information 84 includes second content related information 84A. The second content related information 84A is information related to the content of the advertising video 60. The second content related information 84A is an example of "second content related information" according to the technology of the present disclosure.

[0128] The concept of the content of the advertising video 60 is not limited to the content of the advertising video 60 itself, but also includes the concept of the target audience (e.g., user 14) to whom the advertising video 60 is shown, and / or the attributes of the advertising video 60 (e.g., the product category to which the advertisement shown by the advertising video 60 belongs, the age group targeted by the advertisement shown by the advertising video 60, and / or the age group to which it is ethically safe not to show the advertisement shown by the advertising video 60).

[0129] A first example of the second content-related information 84A is the title of the advertising video 60. The title of the advertising video 60 is decided, for example, by the producer of the advertising video 60. However, this decision method is merely one example, and other decision methods include, for example, a decision method in which the type of subject included in the advertising video 60 is identified by the above-mentioned subject recognition process or the like, and a title suitable for the identified type of subject is derived from the above-mentioned title derivation table or the like.

[0130] A second example of the second content-related information 84A is the name of a sport that the user 14 is watching through the virtual viewpoint video 78 and / or the overhead view video 72. For example, this is determined by the producer of the advertising video 60. However, this determination method is merely one example, and other determination methods include, for example, a determination method in which the type of subject included in the advertising video 60 is identified by the above-mentioned subject recognition process or the like, and the name of a sport that is appropriate for the identified type of subject is derived from the above-mentioned sport name derivation table or the like.

[0131] A third example of the second content-related information 84A is the name of the main subject that the user 14 is looking at through the virtual viewpoint video 78 and / or the bird's-eye view video 72. The name of the main subject is determined, for example, by the producer of the advertising video 60. However, this determination method is merely one example, and other determination methods include, for example, a determination method in which the type of subject included in the advertising video 60 is identified by the above-mentioned subject recognition process or the like, and the name of the main subject that is appropriate for the identified type of subject is derived from the above-mentioned name derivation table or the like.

[0132] In the image processing device 10, the screen data generation unit 28D selects one advertising video 60 to be displayed in the first advertising region 79 (see FIGS. 4, 9, and 10) from the plurality of advertising videos 60 based on the first advertising region-related information 82 (see FIG. 10) and the advertising video-related information 84. In this case, as shown in FIG. 12 as an example, the screen data generation unit 28D selects and acquires, as the advertising video 60A for the first advertising region, the advertising video 60 associated with the advertising video-related information 84 that has the highest degree of match with the first advertising region-related information 82 from the plurality of advertising videos 60 stored in the storage 30. For example, the advertising video 60A for the first advertising region is the advertising video 60 associated with the advertising video-related information 84 including the second content-related information 84A that corresponds to the first content-related information 82A, from the plurality of advertising videos 60 stored in the storage 30. The second content-related information 84A corresponding to the first content-related information 82A refers to, for example, the second content-related information 84A that matches the first content-related information 82A.

[0133] The screen data generation unit 28D generates virtual viewpoint video screen data 80 so that the first advertising area 79 including the advertising video for the first advertising area 60A is displayed on the virtual viewpoint video screen 68. For example, the screen data generation unit 28D generates, as the virtual viewpoint video screen data 80, screen data showing the virtual viewpoint video screen 68 in which the first advertising area 79 in which the advertising video for the first advertising area 60A is displayed is superimposed on the upper right part when viewed from the front.

[0134] 13 as an example, in the image processing device 10, the screen data generation unit 28D outputs virtual viewpoint video screen data 80 to the transmission / reception device 24. The screen data generation unit 28D also outputs the virtual viewpoint video screen data 80 to a storage device such as the storage 30. As a result, the virtual viewpoint video screen data 80 is stored in the storage device such as the storage 30.

[0135] The transmission / reception device 24 transmits the virtual viewpoint video screen data 80 input from the screen data generation unit 28D to the user device 12. In the user device 12, the transmission / reception device 44 receives the virtual viewpoint video screen data 80 transmitted from the image processing device 10. The processor 52 displays the virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 received by the transmission / reception device 44 on the touch panel display 16.

[0136] Next, the operation of the image processing device 10 according to the first embodiment will be described with reference to FIG.

[0137] 14 shows an example of the flow of screen generation processing performed by the processor 28 of the image processing device 10 according to the first embodiment. The flow of screen generation processing shown in FIG. 14 is an example of the "image processing method" according to the technique of the present disclosure.

[0138] 14, first, in step ST10, the reception screen generating unit 28A generates reception screen data 70 based on a plurality of captured images 64 (see FIG. 4). After the processing of step ST10 is executed, the screen generating processing proceeds to step ST12.

[0139] In step ST12, the reception screen generating unit 28A causes the transmitting / receiving device 24 to transmit the generated reception screen data 70 to the user device 12. After the processing of step ST12 is executed, the screen generating processing proceeds to step ST14.

[0140] When the reception screen data 70 is transmitted from the image processing device 10 to the user device 12 by executing the processing of step ST12, the user device 12 receives the reception screen data 70 and displays the reception screen 66 indicated by the received reception screen data 70 on the display 18 (see FIGS. 4 to 6). When the reception screen 66 is displayed on the display 18 of the user device 12, the user 14 gives instructions such as a viewpoint and a gaze point to the user device 12 via the touch panel 20 (see FIGS. 5 and 6). The processor 52 of the user device 12 generates viewpoint information 74 based on the viewpoint and the gaze point received by the touch panel 20, and transmits the generated viewpoint information 74 to the image processing device 10 via the transmission / reception device 44 (see FIG. 7).

[0141] In step ST14, the viewpoint information acquisition unit 28B determines whether or not the viewpoint information 74 has been received by the transmission / reception device 24. If the viewpoint information 74 has not been received by the transmission / reception device 24 in step ST14, the determination is negative, and the screen generation process proceeds to step ST24. If the viewpoint information 74 has been received by the transmission / reception device 24 in step ST14, the determination is positive, and the screen generation process proceeds to step ST16. The viewpoint information acquisition unit 28B acquires the viewpoint information 74 received by the transmission / reception device 24 (see FIG. 7).

[0142] In step ST16, the virtual viewpoint image generation unit 28C generates a virtual viewpoint video 78 based on the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B and the plurality of captured images 64 (see FIG. 8). Then, the screen data generation unit 28D generates virtual viewpoint video screen data 80 that indicates a virtual viewpoint video screen 68 including the virtual viewpoint video 78 generated by the virtual viewpoint image generation unit 28C. After the processing of step ST16 is executed, the screen generation processing proceeds to step ST18.

[0143] In step ST18, the screen data generation unit 28D generates first advertising area related information 82 based on various information, and associates the generated first advertising area related information 82 with the first advertising area 79 (see FIG. 10). After the processing of step ST18 is executed, the screen generation processing proceeds to step ST20.

[0144] In step ST20, the screen data generation unit 28D selects and acquires, from the plurality of advertising videos 60 stored in the storage 30, the advertising video 60 associated with the advertising video related information 84 that has the highest degree of match with the first advertising area related information 82 as the advertising video 60A for the first advertising area (see FIG. 12). After the processing of step ST20 is executed, the screen generation processing proceeds to step ST21.

[0145] In step ST21, the screen data generation unit 28D generates virtual viewpoint video screen data 80 so that the first advertising region 79 including the advertising video for the first advertising region 60A is displayed on the virtual viewpoint video screen 68 (see FIG. 12). That is, the screen data generation unit 28D processes the virtual viewpoint video screen data 80 generated in step ST16 so that the first advertising region 79 including the advertising video for the first advertising region 60A is displayed on the virtual viewpoint video screen 68. After the processing of step ST21 is executed, the screen generation processing proceeds to step ST22.

[0146] In step ST22, the screen data generation unit 28D outputs the virtual viewpoint video screen data 80 generated in step ST22 to the transmission / reception device 24 and a storage device such as the storage 30 (see FIG. 13). As a result, the virtual viewpoint video screen data 80 is stored in a storage device such as the storage 30, and the transmission / reception device 24 transmits the virtual viewpoint video screen data 80 to the user device 12. In the user device 12, the processor 52 displays the virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 transmitted from the transmission / reception device 24 on the touch panel display 16 (see FIG. 13). After the processing of step ST22 is executed, the screen generation processing proceeds to step ST24.

[0147] In step ST24, the screen data generation unit 28D determines whether or not a condition for terminating the screen generation process (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the screen generation process has been accepted by an acceptance device such as the touch panel display 16. If the termination condition has not been satisfied in step ST24, the determination is negative, and the screen generation process proceeds to step ST10. If the termination condition has been satisfied in step ST24, the determination is positive, and the screen generation process terminates.

[0148] As described above in detail, the image processing device 10 outputs virtual viewpoint video screen data 80 for displaying the first advertising area advertising video 60A selected from the plurality of advertising videos 60 in the first advertising area 79 based on the first advertising area related information 82 associated with the first advertising area 79 for the virtual viewpoint video 78 and the advertising video related information 84 associated with each advertising video 60. The virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 is displayed on the touch panel display 16 of the user device 12. The first advertising area advertising video 60A is displayed in the first advertising area 79 within the virtual viewpoint video screen 68. Therefore, according to this configuration, the first advertising area advertising video 60A can be shown to the user 14 watching the virtual viewpoint video 78.

[0149] Furthermore, in the image processing device 10, first advertising area-related information 82 associated with the first advertising area 79 includes first content-related information 82A related to the content of the virtual viewpoint video 78. The advertising video 60A for the first advertising area is selected based on the first content-related information 82A related to the content of the virtual viewpoint video 78 and the advertising video-related information 84 related to the advertising video 60. Therefore, according to this configuration, the advertising video 60A for the first advertising area selected based on the first content-related information 82A related to the content of the virtual viewpoint video 78 and the advertising video-related information 84 related to the advertising video 60 can be shown to the user 14 watching the virtual viewpoint video 78.

[0150] Furthermore, in the image processing device 10, the advertising video related information 84 includes second content-related information 84A related to the content of the advertising video 60. Then, the advertising video 60 associated with the advertising video related information 84 including the second content-related information 84A corresponding to the first content-related information 82A related to the content of the virtual viewpoint video 78 is selected as the advertising video 60A for the first advertising area. Therefore, according to this configuration, the advertising video 60A for the first advertising area selected based on the first content-related information 82A related to the content of the virtual viewpoint video 78 and the second content-related information 84A related to the content of the advertising video 60 can be shown to the user 14 watching the virtual viewpoint video 78.

[0151] Furthermore, in the image processing device 10, the first advertising area related information 82 is changed in accordance with the viewpoint information 74. When the first advertising area related information 82 is changed, there is a high possibility that the advertising video related information 84 that matches the first advertising area related information 82 will also change. A change in the advertising video related information 84 that matches the first advertising area related information 82 means that the advertising video 60 displayed in the first advertising area 79 will also change. Therefore, according to this configuration, the advertising video 60 to be displayed in the first advertising area 79 can be changed by changing the viewpoint information 74 required to generate the virtual viewpoint video 78.

[0152] [Second embodiment] In the second embodiment, the same components as those described in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted, and only the portions different from the first embodiment will be described.

[0153] 15 as an example, the virtual viewpoint video screen 68 has a second advertising area 86 in addition to a first advertising area 79. That is, in the virtual viewpoint video 78, the second advertising area 86 is displayed in addition to the first advertising area 79 as an area related to the virtual viewpoint video 78. The second advertising area 86 is an example of a "second area" according to the technology of the present disclosure.

[0154] In the example shown in FIG. 15, the second advertising area 86 is rectangular and is displayed in the lower right corner of the virtual viewpoint video screen 68 when viewed from the front. The size of the second advertising area 86 is smaller than that of the first advertising area 79. Note that the second advertising area 86 may be lower than the first advertising area 79 not only in size but also in, for example, resolution and / or transparency. In other words, the specifications of the first advertising area 79 and the second advertising area 86 may be different so that the first advertising area 79 is more likely to attract attention and be more visible to users 14 than the second advertising area 86. As will be described in more detail below, an advertising video 60 (see FIGS. 16 to 18) is displayed in the second advertising area 86.

[0155] The second advertising area 86 is an area related to the virtual viewpoint video 78. The second advertising area 86 is displayed superimposed on the virtual viewpoint video 78 in the same manner as the first advertising area 79. Here, an example in which the second advertising area 86 is displayed superimposed on the virtual viewpoint video 78 is described, but this is merely an example, and the second advertising area 86 may be displayed embedded in the virtual viewpoint video 78.

[0156] The screen data generation unit 28D generates second advertising area related information 88 based on various information. The second advertising area related information 88 is information associated with the second advertising area 86. The various information used to generate the second advertising area related information 88 is the same as the various information used to generate the first advertising area related information 82, for example. The second advertising area related information 88 is an example of the "second information" and "area related information" according to the technology of the present disclosure.

[0157] The second advertising area related information 88 includes third content related information 88A. The third content related information 88A is information related to the content of the virtual viewpoint video 78. An example of the third content related information 88A is the same information as the first content related information 82A.

[0158] Furthermore, when the third content-related information 88A is determined according to the type of subject identified by the subject recognition process in the same manner as the first content-related information 82A, the type of subject identified by the screen data generation unit 28D is changed according to the viewpoint information 74 required to generate the virtual viewpoint video 78, and therefore the third content-related information 88A is also changed accordingly. For example, the third content-related information 88A is changed by the screen data generation unit 28D according to at least one of the total time information 74A, viewpoint path information 74B, required time information 74C, elapsed time information 74D, movement speed information 74E, angle of view information 74F, and point of interest information 74G included in the viewpoint information 74.

[0159] The screen data generation unit 28D associates the generated second advertising area related information 88 with the second advertising area 86. Note that information associating the second advertising area 86 with the second advertising area related information 88 may be stored in a storage device such as the storage 30 together with the virtual viewpoint video 78 or separately from the virtual viewpoint video 78.

[0160] As an example, as shown in FIG. 16, the plurality of pieces of advertising video-related information 84 stored in the storage 30 include a rank identifier 90 in addition to the second content-related information 84A. The rank identifier 90 is an identifier indicating the rank of the advertising video 60. The rank identifier 90 is categorized, for example, into rank A and rank B. The rank identifier 90 is determined by the producer of the advertising video 60. However, this determination method is merely an example, and other determination methods include, for example, a determination method in which the type of subject included in the advertising video 60 is identified by the above-mentioned subject recognition process or the like, and a rank identifier 90 appropriate for the identified type of subject is derived using a rank derivation table (not shown) or the like. The rank derivation table is, for example, a table in which the type of subject is input and the rank identifier is output.

[0161] The A rank is an identifier that instructs that the advertising video 60 associated with the advertising video related information 84 including the rank identifier 90 indicating the A rank is to be displayed in the first advertising area 79, and the B rank is an identifier that instructs that the advertising video 60 associated with the advertising video related information 84 including the rank identifier 90 indicating the B rank is to be displayed in the second advertising area 86. Whether the rank identifier 90 is set to A rank or B rank is determined, for example, by the producer of the advertising video 60 in accordance with the charge of the party (e.g., advertiser) that provides the advertising video 60 to the user 14. Charge means, for example, that the producer of the advertising video 60 or the producer of the image processing system 2 charges a fee to the party that provides the advertising video 60 to the user 14. The rank identifier 90 is information that indicates charge. The rank identifier 90 is an example of "charge information" related to the technology of the present disclosure.

[0162] Here, the content of the rank identifier 90 (i.e., the rank) is determined based on the charge, but the content of the rank identifier 90 may also be determined based on criteria other than the charge (for example, whether or not a company that supports a soccer home game is the provider of the advertising video 60).

[0163] In the image processing device 10, the screen data generation unit 28D selects the advertising video 60A for the first advertising area in the manner described in the first embodiment, and also selects one advertising video 60 to be displayed in the second advertising area 86 (see FIGS. 15, 17, and 18) from the multiple advertising videos 60 based on the second advertising area-related information 88 (see FIG. 16) and the advertising video-related information 84. In this case, as shown in FIG. 17 as an example, the screen data generation unit 28D selects and acquires, as the advertising video 60B for the second advertising area, the advertising video 60 associated with the advertising video-related information 84 that has the highest degree of match with the second advertising area-related information 88 from the multiple advertising videos 60 stored in the storage 30. For example, the advertising video 60B for the second advertising area is the advertising video 60 associated with the advertising video-related information 84 including the second content-related information 84A that corresponds to the third content-related information 88A, from the multiple advertising videos 60 stored in the storage 30. The second content-related information 84A corresponding to the third content-related information 88A refers to, for example, the second content-related information 84A that matches the third content-related information 88A.

[0164] The screen data generation unit 28D generates the virtual viewpoint video screen data 80 so that the first advertising area 79 including the advertising video for the first advertising area 60A is displayed on the virtual viewpoint video screen 68, and the second advertising area 86 including the advertising video for the second advertising area 60B is displayed on the virtual viewpoint video screen 68. For example, the screen data generation unit 28D generates, as the virtual viewpoint video screen data 80, screen data indicating the virtual viewpoint video screen 68 in which the first advertising area 79 in which the advertising video for the first advertising area 60A is displayed is superimposed on the upper right corner when viewed from the front, and the second advertising area 86 in which the advertising video for the second advertising area 60B is displayed is superimposed on the lower right corner when viewed from the front.

[0165] 18 , the virtual viewpoint video screen data 80 generated by the screen data generating unit 28D as described above is stored in a storage device such as the storage 30, and is transmitted from the image processing device 10 to the user device 12, in the same manner as described in the first embodiment. Then, in the user device 12, the virtual viewpoint video screen data 80 transmitted from the image processing device 10 is received by the transmitting / receiving device 44. The processor 52 causes the touch panel display 16 to display the virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 received by the transmitting / receiving device 44.

[0166] In the image processing device 10, the screen data generation unit 28D outputs the virtual viewpoint video screen data 80 to the transmission / reception device 24. The screen data generation unit 28D also outputs the virtual viewpoint video screen data 80 to a storage device such as the storage 30. As a result, the virtual viewpoint video screen data 80 is stored in the storage device such as the storage 30. In the second embodiment, the virtual viewpoint video screen data 80 is an example of the "second data" according to the technology of the present disclosure.

[0167] The transmitting / receiving device 24 transmits the virtual viewpoint video screen data 80 input from the screen data generating unit 28D to the user device 12. In the user device 12, the transmitting / receiving device 44 receives the virtual viewpoint video screen data 80 transmitted from the image processing device 10. The processor 52 displays a virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 received by the transmitting / receiving device 44 on the touch panel display 16. On the virtual viewpoint video screen 68, a first advertising area 79 including the advertising video for the first advertising area 60A and a second advertising area 86 including the advertising video for the second advertising area 60B are displayed side by side on the same screen.

[0168] Next, the operation of the image processing device 10 according to the second embodiment will be described with reference to FIGS. 19A and 19B.

[0169] 19A and 19B show an example of the flow of screen generation processing performed by the processor 28 of the image processing device 10 according to the second embodiment. The flow of screen generation processing shown in FIGS. 19A and 19B is an example of an "image processing method" according to the technology of the present disclosure. The flowcharts shown in FIGS. 19A and 19B differ from the flowchart shown in FIG. 14 in that they include processing of step ST100 instead of processing of step ST18, and in that they include processing of steps ST102 to ST106 instead of processing of step ST21. In the second embodiment, only steps that differ from the flowchart shown in FIG. 14 will be described below.

[0170] 19A, in step ST100, the screen data generation unit 28D generates first advertising area related information 82 based on various information, and associates the generated first advertising area related information 82 with the first advertising area 79 (see FIG. 10). Furthermore, the screen data generation unit 28D generates second advertising area related information 88 based on various information, and associates the generated second advertising area related information 88 with the second advertising area 86 (see FIG. 16). After the processing of step ST100 is executed, the screen generation processing proceeds to step ST102.

[0171] 19B, the screen data generation unit 28D generates the virtual viewpoint video screen data 80 so that the second advertising area 86 including the advertising video for the second advertising area 60B is displayed on the virtual viewpoint video screen 68 (see FIG. 17). That is, the screen data generation unit 28D processes the virtual viewpoint video screen data 80 generated in step ST16 so that the second advertising area 86 including the advertising video for the second advertising area 60B is displayed on the virtual viewpoint video screen 68. After the processing of step ST102 is executed, the screen generation processing proceeds to step ST104.

[0172] In step ST106, the screen data generation unit 28D outputs the virtual viewpoint video screen data 80 generated in step ST104 to the transmission / reception device 24 and a storage device such as the storage 30 (see FIG. 18). As a result, the virtual viewpoint video screen data 80 is stored in a storage device such as the storage 30, and the transmission / reception device 24 transmits the virtual viewpoint video screen data 80 to the user device 12. In the user device 12, the processor 52 displays the virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 transmitted from the transmission / reception device 24 on the touch panel display 16 (see FIG. 18). After the processing of step ST106 is executed, the screen generation processing proceeds to step ST24.

[0173] As described in detail above, the image processing device 10 outputs virtual viewpoint video screen data 80 for displaying the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 in the first advertising area 79 based on first advertising area related information 82 associated with the first advertising area 79 for the virtual viewpoint video 78 and advertising video related information 84 associated with each advertising video 60.

[0174] In addition, the image processing device 10 outputs virtual viewpoint video screen data 80 for displaying an advertising video 60B for the second advertising area selected from the multiple advertising videos 60 in the second advertising area 86 based on second advertising area related information 88 associated with the second advertising area 86 related to the virtual viewpoint video 78 and advertising video related information 84 related to the advertising video 60 associated with each advertising video 60.

[0175] The virtual viewpoint video screen 68 indicated by the virtual viewpoint video screen data 80 is displayed on the touch panel display 16 of the user device 12. The advertising video for the first advertising area 60A is displayed in a first advertising area 79 within the virtual viewpoint video screen 68, and the advertising video for the second advertising area 60B is displayed in an advertising area 86 within the virtual viewpoint video screen 68 (see FIG. 18 ). Therefore, according to this configuration, the advertising video for the first advertising area 60A and the advertising video for the second advertising area 60B can be shown to the user 14 viewing the virtual viewpoint video 78.

[0176] Furthermore, in the image processing device 10, the second advertising area related information 88 is changed in accordance with the viewpoint information 74. When the second advertising area related information 88 is changed, there is a high possibility that the advertising video related information 84 that matches the second advertising area related information 88 will also change. A change in the advertising video related information 84 that matches the second advertising area related information 88 means that the advertising video 60 displayed in the second advertising area 86 will also change. Therefore, according to this configuration, the advertising video 60 displayed in the second advertising area 86 can be changed by changing the viewpoint information 74 required to generate the virtual viewpoint video 78.

[0177] Furthermore, in the image processing device 10, the plurality of pieces of advertising video-related information 84 stored in the storage 30 include a rank identifier 90 in addition to the second content-related information 84A. The content of the rank identifier 90, i.e., the rank given to the advertising video 60, is determined based on the charges of the party providing the advertising video 60 to the user 14. Therefore, with this configuration, the advertising video 60 displayed in the first advertising area 79 and the second advertising area 86 can be changed according to the charges of the party providing the advertising video 60 to the user 14.

[0178] [First Modification] Next, a first modified example of the image processing device 10 will be described.

[0179] 20, the first advertising area related information 82 further includes first advertising effect related information 82B. The first advertising effect related information 82B is information related to advertising effect (for example, advertising effect when advertising video 60 is displayed in the first advertising area 79). The second advertising area related information 88 further includes second advertising effect related information 88B. The second advertising effect related information 88B is information related to advertising effect (for example, advertising effect when advertising video 60 is displayed in the second advertising area 86).

[0180] In this first modified example, the advertising effect is, for example, either "large" or "small." A "large" advertising effect indicates a greater advertising effect than a "small" advertising effect. Whether the advertising effect identified from the first advertising effect related information 82B is large or not, and whether the advertising effect identified from the second advertising effect related information 88B is large or not, are determined, for example, by the producer of the advertising video 60 and / or the producer of the image processing system 2. Note that, in this first modified example, the advertising effect can also be expressed on a finer scale than "large" and "small." The magnitude of the advertising effect also changes depending on the viewpoint information 74.

[0181] 21 as an example, the advertising video related information 84 further includes third advertising effect related information 84B. The third advertising effect related information 84B is information related to advertising effect (for example, advertising effect when the advertising video 60 associated with the advertising video related information 84 is displayed). Whether the advertising effect identified from the third advertising effect related information 84B is large or small is determined by, for example, the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2, etc.

[0182] As described above, in the image processing device 10 according to the first modified example, the first advertising effect-related information 82B is included in the first advertising area-related information 82 as information relating to advertising effect (see FIG. 20). Furthermore, the advertising video-related information 84 includes the third advertising effect-related information 84B as information relating to advertising effect to be compared with the first advertising effect-related information 82B when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 21). Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 based on the advertising effect can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79.

[0183] Note that instead of the third advertising effect-related information 84B, the advertising video 60A for the first advertising area may be selected based on the first advertising effect-related information 82B and the rank determined by the rank identifier 90. For example, if the advertising effect of the first advertising effect-related information 82B is "high," the advertising video 60 assigned a rank identifier of A may be displayed in the first advertising area 79. In other words, an advertisement with a high billing amount may be displayed in an area with high advertising effectiveness.

[0184] Furthermore, in the image processing device 10 according to the first modified example, second advertising effect-related information 88B is included in the second advertising area-related information 88 as information relating to advertising effect (see FIG. 20). Furthermore, third advertising effect-related information 84B is included in the advertising video-related information 84 as information relating to advertising effect to be compared with the second advertising effect-related information 88B when determining the degree of coincidence between the second advertising area-related information 88 and the advertising video-related information 84 (see FIG. 21). Therefore, according to this configuration, the advertising video 60B for the second advertising area selected from the multiple advertising videos 60 based on the advertising effect can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86.

[0185] Also, in the second advertising area 86, similarly to the first advertising area 79, the advertising video 60B for the second advertising area may be selected based on the second advertising effect related information 88B and the rank determined by the rank identifier 90, instead of the third advertising effect related information 84B. For example, if the advertising effect is "low" in the second advertising effect related information 88B, the advertising video 60 assigned a rank identifier of B rank may be displayed in the second advertising area 86. In other words, an advertisement with a low billing amount may be displayed in an area with low advertising effect.

[0186] Furthermore, the first advertising effect-related information 82B and the second advertising effect-related information 88B may be changed based on the viewpoint information 74. When the first advertising area 79 and the second advertising area 86 are embedded and displayed in the virtual viewpoint video 78, the positions, sizes, and / or orientations of the first advertising area 79 and the second advertising area 86 are changed depending on the viewpoint position and / or direction determined by the viewpoint information 74. Therefore, the advertising effects of the first advertising area 79 and the second advertising area 86 are changed according to the viewpoint information 74. In this manner, the first advertising effect-related information 82B and the second advertising effect-related information 88B may be determined taking into account the advertising effects determined by the viewpoint information 74. In this case, for example, an advertisement with a high billing amount is displayed in the second advertising area 86. For example, if the viewpoint position is such that the second advertising area 86 is located closer to the center than the first advertising area 79, the advertising effect of the second advertising effect-related information 88B may be set to "high" and the advertising effect of the first advertising effect-related information 82B may be set to "low."

[0187] [Second Modification] Next, a second modified example of the image processing device 10 will be described.

[0188] As an example, as shown in FIG. 22, the first advertising area related information 82 further includes first size related information 82C. The first size related information 82C is information relating to the size at which the advertising video 60 is to be displayed in the first advertising area 79. Furthermore, the second advertising area related information 88 further includes second size related information 88C. The second size related information 88C is information relating to the size at which the advertising video 60 is to be displayed in the second advertising area 86. In this second modified example, the size identified from the second size related information 88C is, for example, either "large" or "small." The size "large" indicates a larger size than the size "small."

[0189] The first size-related information 82C and the second size-related information 88C may be two-dimensional or three-dimensional. For example, when the first advertising area 79 is displayed superimposed on the virtual viewpoint video 78, the first size-related information 82C and the second size-related information 88C may be two-dimensional, and when the first advertising area 79 is displayed embedded in the virtual viewpoint video 78, the first size-related information 82C and the second size-related information 88C may be three-dimensional.

[0190] As an example, as shown in Fig. 23, the advertising video related information 84 further includes third size related information 84C. The third size related information 84C is information relating to the size of the advertising video 60 (for example, the size displayed on the screen). In this second modified example, the size identified from the third size related information 84C is, for example, either "large" or "small." The size "large" indicates a size larger than the size "small."

[0191] The third size-related information 84C may be a two-dimensional size or a three-dimensional size. For example, if the advertising video 60 is a two-dimensional image, the third size-related information 84C may be a two-dimensional size, or if the advertising video 60 is a virtual viewpoint image, the third size-related information 84C may be a three-dimensional size.

[0192] As described above, in the image processing device 10 according to the second modified example, the first advertising area-related information 82 includes first size-related information 82C as information relating to the size at which the advertising video 60 is displayed in the first advertising area 79 (see FIG. 22). Furthermore, the advertising video-related information 84 includes third size-related information 84C as information relating to the size to be compared with the first size-related information 82C when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 23). Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the plurality of advertising videos 60 based on the size at which the advertising video 60 is displayed in the first advertising area 79 and the size of the advertising video 60 can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79.

[0193] Furthermore, in the image processing device 10 according to the second modified example, second size-related information 88C is included in the second advertising area-related information 88 as information relating to the size at which the advertising video 60 is displayed in the second advertising area 86 (see FIG. 22). Furthermore, third size-related information 84C is included in the advertising video-related information 84 as information relating to the size to be compared with the second size-related information 88C when determining the degree of coincidence between the second advertising area-related information 88 and the advertising video-related information 84 (see FIG. 23). Therefore, according to this configuration, the advertising video 60B for the second advertising area selected from the plurality of advertising videos 60 based on the size at which the advertising video 60 is displayed in the second advertising area 86 and the size of the advertising video 60 can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86.

[0194] Furthermore, the first size-related information 82C and the second size-related information 88C may be changed based on the viewpoint information 74. When the first advertising area 79 and the second advertising area 86 are embedded and displayed in the virtual viewpoint video 78, the sizes of the first advertising area 79 and the second advertising area 86 are changed depending on the viewpoint position and / or direction, etc., determined by the viewpoint information 74. In this way, the first size-related information 82C and the second size-related information 88C may be determined taking into account the display size determined by the viewpoint information 74. For example, depending on the viewpoint position, the second advertising area 86 is displayed in a size larger than the first advertising area 79. In this case, the display size specified by the first size-related information 82C is "small," the display size specified by the second size-related information 88C is "large," and an advertisement whose size specified by the third size-related information 84C is "large" is displayed in the second advertising area 86.

[0195] [Third Modification] Next, a third modified example of the image processing device 10 will be described.

[0196] 24 as an example, the first advertising area-related information 82 further includes first viewpoint information 82D. The first viewpoint information 82D is viewpoint information 74 required to generate a virtual viewpoint video 78. The second advertising area-related information 88 further includes second viewpoint information 88D. Like the first viewpoint information 82D, the second viewpoint information 88D is also viewpoint information 74 required to generate the virtual viewpoint video 78. In other words, once the viewpoint information 74 is determined, the appearance of the first advertising area 79 and the second advertising area 86 is determined, and therefore once the viewpoint information 74 is determined, the first viewpoint information 82D and the second viewpoint information 88D are also determined.

[0197] 25 as an example, the advertising video related information 84 further includes third viewpoint information 84D. The technical meaning of the third viewpoint information 84D is the same as that of the viewpoint information 74. The third viewpoint information 84D is viewpoint information used to generate a virtual viewpoint video, and is determined, for example, by the party that provides the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2, etc.

[0198] As described above, in the image processing device 10 according to the third modified example, the first viewpoint information 82D is included in the first advertising area-related information 82 (see FIG. 24). Furthermore, the advertising video-related information 84 includes third viewpoint information 84D that is compared with the first viewpoint information 82D when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 25). Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 based on the first viewpoint information 82D included in the first advertising area-related information 82 and the third viewpoint information 84D included in the advertising video-related information 84 can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79.

[0199] For example, if the advertising video 60 is a virtual viewpoint video, the advertising video 60 viewed from the same viewpoint position as the virtual viewpoint video 78 can be displayed in the first advertising area 79. This allows the user 14 to view the advertising video 60 without feeling uncomfortable. Note that the first viewpoint information 82D and the third viewpoint information 84D do not need to be completely identical, but it is desirable that they are similar enough that the user 14 does not feel uncomfortable. Furthermore, if the third viewpoint information 84D is different from the viewpoint information 74, the viewpoint position for viewing the advertising video 60, which is a virtual viewpoint video, may be changed so that the third viewpoint information 84D matches the viewpoint information 74, and the advertising video 60 may be displayed in the first advertising area 79. For example, if the user changes the viewpoint information 74, the display of the advertising video 60A for the first advertising area may also be changed and displayed in the same way.

[0200] Moreover, in the image processing device 10 according to the third modified example, the second viewpoint information 88D is included in the second advertising area-related information 88 (see FIG. 24). Moreover, the advertising video-related information 84 includes third viewpoint information 84D, which is compared with the second viewpoint information 88D when determining the degree of coincidence between the second advertising area-related information 88 and the advertising video-related information 84 (see FIG. 25). Therefore, according to this configuration, the second advertising area advertising video 60B selected from the multiple advertising videos 60 based on the second viewpoint information 88D included in the second advertising area-related information 88 and the third viewpoint information 84D included in the advertising video-related information 84 can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86. The second advertising area advertising video 60B may be modified and displayed in the same manner as the first advertising area advertising video 60A.

[0201] The first viewpoint information 82D may include first viewpoint path information equivalent to the viewpoint path information 74B as information relating to the first viewpoint path (for example, viewpoint path P1). The second viewpoint information 88D may also include second viewpoint path information equivalent to the viewpoint path information 74B as information relating to the second viewpoint path (for example, viewpoint path P1). Furthermore, the third viewpoint information 84D may also include third viewpoint path information as information relating to the third viewpoint path (for example, a viewpoint path determined by the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2, etc.).

[0202] Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the plurality of advertising videos 60 based on the first viewpoint path information included in the first viewpoint information 82D and the third viewpoint path information included in the third viewpoint information 84D can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79. Furthermore, according to this configuration, the advertising video 60B for the second advertising area selected from the plurality of advertising videos 60 based on the second viewpoint path information included in the second viewpoint information 88D and the third viewpoint path information included in the third viewpoint information 84D can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86.

[0203] In addition, when the advertising video 60 is a virtual viewpoint video, the advertising video 60A for the first advertising area or the advertising video 60B for the second advertising area may be displayed by changing the third viewpoint path information in the same manner as the first viewpoint path information or the second viewpoint path information.

[0204] [Fourth Modification] Next, a fourth modified example of the image processing device 10 will be described.

[0205] 26, the first advertising area-related information 82 further includes first display time-related information 82E. The first display time-related information 82E is information related to the time during which the first advertising area 79 is displayed (for example, the time during which the first advertising area 79 is continuously displayed on the virtual viewpoint video screen 68). The second advertising area-related information 88 further includes second display time-related information 88E. The second display time-related information 88E is information related to the time during which the second advertising area 86 is displayed (for example, the time during which the second advertising area 86 is continuously displayed on the virtual viewpoint video screen 68). In the fourth modified example, the time specified from the first display time-related information 82E and / or the time specified from the second display time-related information 88E is a time determined by, for example, the producer of the advertising video 60 and / or the producer of the image processing system 2. For example, once the viewpoint information 74 is determined, the time during which the first advertising area 79 and the second advertising area 86 are displayed is determined.

[0206] 27, the advertising video related information 84 further includes third display time related information 84E. The third display time related information 84E is a predetermined time for which the advertising video 60 is displayed (for example, the time for which the advertising video 60 is continuously displayed). In the fourth modified example, the time specified from the third display time related information 84E is a time determined by, for example, the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2.

[0207] As described above, in the image processing device 10 according to the fourth modification, the first advertising area-related information 82 includes first display time-related information 82E as information relating to the time for which the advertising video 60 is displayed in the first advertising area 79 (see FIG. 26). Furthermore, the advertising video-related information 84 includes third display time-related information 84E as information relating to the time to be compared with the first display time-related information 82E when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 27). Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 based on the time for which the advertising video 60 is displayed in the first advertising area 79 and the predetermined time for which the advertising video 60 is displayed can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79.

[0208] Furthermore, in the fourth modified example, an example of a time specified from the first display time related information 82E is the time during which the first advertising area 79 is continuously displayed on the virtual viewpoint video screen 68, and an example of a time specified from the third display time related information 84E is the time during which the advertising video 60 is continuously displayed. Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 based on the time during which the advertising video 60 is continuously displayed in the first advertising area 79 and the predetermined time period for which the advertising video 60 is continuously displayed can be shown to the user 14 watching the virtual viewpoint video 78 via the first advertising area 79.

[0209] Furthermore, in the image processing device 10 according to the fourth modification, second display time related information 88E is included in the second advertising area related information 88 as information relating to the time for which the advertising video 60 is displayed in the second advertising area 86 (see FIG. 26). Furthermore, third display time related information 84E is included in the advertising video related information 84 as information relating to the time to be compared with the second display time related information 88E when determining the degree of coincidence between the second advertising area related information 88 and the advertising video related information 84 (see FIG. 27). Therefore, according to this configuration, the advertising video 60B for the second advertising area selected from the plurality of advertising videos 60 based on the time for which the second advertising area 86 is displayed and the time predetermined as the time for which the advertising video 60 is displayed can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86.

[0210] Furthermore, in the fourth modified example, an example of a time specified from the second display time related information 88E is the time during which the second advertising area 86 is continuously displayed on the virtual viewpoint video screen 68, and an example of a time specified from the third display time related information 84E is the time during which the advertising video 60 is continuously displayed. Therefore, according to this configuration, the second advertising area advertising video 60B selected from the multiple advertising videos 60 based on the time during which the second advertising area 86 is continuously displayed and the predetermined time during which the advertising video 60 is continuously displayed can be shown to the user 14 watching the virtual viewpoint video 78 via the second advertising area 86.

[0211] In this fourth variant, an example of a time determined from the first display time related information 82E is the time during which the first advertising area 79 is continuously displayed on the virtual viewpoint video screen 68, but this is merely an example, and the time determined from the first display time related information 82E may also be the time during which the first advertising area 79 is intermittently displayed on the virtual viewpoint video screen 68.

[0212] Furthermore, in this fourth variant, the time during which the second advertising area 86 is continuously displayed on the virtual viewpoint video screen 68 is given as an example of the time determined from the second display time related information 88E, but this is merely an example, and the time determined from the second display time related information 88E may also be the time during which the second advertising area 86 is intermittently displayed on the virtual viewpoint video screen 68.

[0213] [Fifth Modification] Next, a fifth modified example of the image processing device 10 will be described.

[0214] 28, the advertising video related information 84 further includes total playback time information 84F. The total playback time information 84F is information indicating the total playback time. The total playback time refers to a predetermined time required from the start to the end of playback of the advertising video 60 (for example, the time from the start to the end of playback when played at standard speed without double speed playback). The total playback time is a time determined, for example, by the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2.

[0215] 29 as an example, the screen data generation unit 28D selects and acquires, from the plurality of advertising videos 60 stored in the storage 30, an advertising video 60 whose first display time is equal to or longer than the total playback time indicated by the total playback time information 84F and which is associated with advertising video related information 84 that has the highest degree of match with the first advertising area related information 82, as the advertising video 60A for the first advertising area. Here, the first display time refers to the time specified from the first display time related information 82E included in the first advertising area related information 82.

[0216] Furthermore, the screen data generation unit 28D selects and acquires, from the plurality of advertising videos 60 stored in the storage 30, an advertising video 60 whose second display time is equal to or longer than the total playback time indicated by the total playback time information 84F and which is associated with advertising video related information 84 that has the highest degree of match with the second advertising area related information 88, as the advertising video 60B for the second advertising area. Here, the second display time refers to the time specified from the second display time related information 88E included in the second advertising area related information 88.

[0217] The screen data generation unit 28D generates the virtual viewpoint video screen data 80 based on the first display time related information 82E and the total playback time information 84F. The screen data generation unit 28D also generates the virtual viewpoint video screen 68 based on the second display time related information 88E and the total playback time information 84F.

[0218] That is, the screen data generation unit 28D generates virtual viewpoint video screen data 80 based on the advertising video 60A for the first advertising area selected and acquired from storage 30 based on the first display time related information 82E and the total playback time information 84F, and the advertising video 60B for the second advertising area selected and acquired from storage 30 based on the second display time related information 88E and the total playback time information 84F.

[0219] In this way, in the image processing device according to the fifth modified example, the advertising video 60A for the first advertising area is selected based on the first display time related information 82E and the total playback time information 84F, and the virtual viewpoint video screen data 80 is generated based on the advertising video 60A for the first advertising area. Therefore, according to this configuration, the advertising video 60A for the first advertising area obtained as a moving image taking into consideration the first display time related information 82E and the total playback time information 84F can be shown to the user 14 watching the virtual viewpoint video 78 via the first advertising area 79.

[0220] For example, the advertising video 60 that ends within the time period in which the first advertising area 79 is continuously displayed is displayed in the first advertising area 79. This allows the advertising video 60 to be viewed continuously from start to finish for the user 14.

[0221] Furthermore, in the image processing device according to the fifth modified example, the advertising video 60B for the second advertising area is selected based on the second display time related information 88E and the total playback time information 84F, and the virtual viewpoint video screen data 80 is generated based on the advertising video 60B for the second advertising area. Therefore, according to this configuration, the advertising video 60B for the second advertising area obtained as a moving image taking into consideration the second display time related information 88E and the total playback time information 84F can be shown to the user 14 watching the virtual viewpoint video 78 via the second advertising area 86. A specific example is similar to the advertising video 60A for the first advertising area.

[0222] [Sixth Modification] Next, a sixth modified example of the image processing device 10 will be described.

[0223] 30, the first advertising area-related information 82 further includes first movement speed-related information 82F. The first movement speed-related information 82F is information related to the movement speed of the first viewpoint (for example, a viewpoint included in the viewpoint path P1) required to generate the virtual viewpoint video 78. An example of the movement speed of the first viewpoint is the average value of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74. Note that the average value is merely an example, and may be the movement speed identified from the movement speed information 74E assigned to any one viewpoint, the median of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74, or the mode of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74, etc.

[0224] Furthermore, the second advertising area related information 88 further includes second movement speed related information 88F. The second movement speed related information 88F is information related to the movement speed of a second viewpoint (for example, a viewpoint included in the viewpoint path P1) required to generate the virtual viewpoint video 78. An example of the movement speed of the second viewpoint is the average value of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74. Note that the average value is merely an example, and may be the movement speed identified from the movement speed information 74E assigned to any one viewpoint, the median of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74, or the mode of the movement speeds identified from the movement speed information 74E assigned to the multiple viewpoints included in the viewpoint information 74, etc.

[0225] The movement speed identified from the second movement speed related information 88F and / or the time identified from the second display time related information 88E are times determined, for example, by the producer of the advertising video 60 and / or the producer of the image processing system 2.

[0226] As an example, as shown in FIG. 31 , the advertising video related information 84 further includes third movement speed related information 84G. The third movement speed related information 84G is information related to a movement speed predetermined as the movement speed of the third viewpoint required to generate the virtual viewpoint video. The technical meaning of the third viewpoint is the same as that of the first viewpoint and the second viewpoint. The technical meaning of the movement speed of the third viewpoint is also the same as that of the first viewpoint and the second viewpoint. The third movement speed related information 84G is, for example, a time determined by the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2, etc.

[0227] As described above, in the image processing device 10 according to the sixth modification, the first advertising area-related information 82 includes first movement speed-related information 82F as information related to the movement speed of the first viewpoint required to generate the virtual viewpoint video 78 (see FIG. 30). Furthermore, the advertising video-related information 84 includes third movement speed-related information 84G as information related to the movement speed to be compared with the first movement speed-related information 82F when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 31). Therefore, according to this configuration, the advertising video 60A for the first advertising area selected from the multiple advertising videos 60 based on the movement speed of the first viewpoint and a movement speed predetermined as the movement speed of the third viewpoint required to generate the virtual viewpoint video can be shown to the user 14 viewing the virtual viewpoint video 78 via the first advertising area 79.

[0228] Furthermore, in the image processing device 10 according to the sixth modified example, second movement speed-related information 88F is included in the second advertising area-related information 88 as information related to the movement speed of the second viewpoint required to generate the virtual viewpoint video 78 (see FIG. 30). Furthermore, third movement speed-related information 84G is included in the advertising video-related information 84 as information related to the movement speed to be compared with the second movement speed-related information 88F when determining the degree of coincidence between the second advertising area-related information 88 and the advertising video-related information 84 (see FIG. 31). Therefore, according to this configuration, the advertising video 60B for the second advertising area selected from the multiple advertising videos 60 based on the movement speed of the second viewpoint and a movement speed predetermined as the movement speed of the third viewpoint required to generate the virtual viewpoint video can be shown to the user 14 viewing the virtual viewpoint video 78 via the second advertising area 86.

[0229] [Seventh Modification] Next, a seventh modification of the image processing device 10 will be described.

[0230] As an example, as shown in FIG. 32 , the first advertising area-related information 82 further includes first timing-related information 82G. The first timing-related information 82G is information related to the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 (in the example shown in FIG. 32 , the virtual viewpoint video screen 68) (for example, the timing at which the first advertising area 79 is superimposed and displayed on the virtual viewpoint video 78). A first example of the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 starts. A second example of the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 is several seconds (for example, 5 seconds) after the display of the virtual viewpoint video 78 starts. A third example of the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 ends. A fourth example of the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 is several seconds (for example, 5 seconds) before the timing at which the display of the virtual viewpoint video 78 ends. The timing at which the first advertising region 79 is included in the virtual viewpoint video 78 is determined by, for example, the producer of the advertising video 60 and / or the producer of the image processing system 2.

[0231] The second advertising area-related information 88 further includes second timing-related information 88G. The second timing-related information 88G is information related to the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 (the virtual viewpoint video screen 68 in the example shown in FIG. 32 ) (for example, the timing at which the second advertising area 86 is displayed superimposed on the virtual viewpoint video 78). A first example of the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 starts. A second example of the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 is several seconds (for example, 5 seconds) after the display of the virtual viewpoint video 78 starts. A third example of the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 ends. A fourth example of the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 is several seconds (for example, 5 seconds) before the timing at which the display of the virtual viewpoint video 78 ends. The timing at which the second advertising region 86 is included in the virtual viewpoint video 78 is determined by, for example, the producer of the advertising video 60 and / or the producer of the image processing system 2.

[0232] As an example, as shown in FIG. 33 , the advertising video related information 84 further includes third timing related information 84H. The third timing related information 84H is information related to the timing (e.g., the timing at which the advertising video 60 is displayed in the virtual viewpoint video 78) at which the advertising video 60 is included in the virtual viewpoint video 78 (in the example shown in FIG. 33 , the virtual viewpoint video screen 68). A first example of the timing at which the advertising video 60 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 starts. A second example of the timing at which the advertising video 60 is included in the virtual viewpoint video 78 is several seconds (e.g., 5 seconds) after the display of the virtual viewpoint video 78 starts. A third example of the timing at which the advertising video 60 is included in the virtual viewpoint video 78 is the timing at which the display of the virtual viewpoint video 78 ends. A fourth example of the timing at which the advertising video 60 is included in the virtual viewpoint video 78 is several seconds (e.g., 5 seconds) before the display of the virtual viewpoint video 78 ends. The third timing-related information 84H is, for example, a time determined by the party that provides the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2, or the like.

[0233] As described above, in the image processing device 10 according to the seventh modification, the first timing-related information 82G is included in the first advertising area-related information 82 as information related to the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 (see FIG. 32). In addition, the third timing-related information 84H is included in the advertising video-related information 84 as information related to the timing to be compared with the first timing-related information 82G when determining the degree of coincidence between the first advertising area-related information 82 and the advertising video-related information 84 (see FIG. 33). Therefore, according to this configuration, an advertising video 60A for the first advertising area (for example, an advertising video 60 associated with advertising video related information 84 including third timing related information 84H that coincides with the timing when the first advertising area 79 is included in the virtual viewpoint video 78 and the timing when the advertising video 60 is included in the virtual viewpoint video 78) selected from a plurality of advertising videos 60 based on the timing when the first advertising area 79 is included in the virtual viewpoint video 78 and the timing when the advertising video 60 is included in the virtual viewpoint video 78 can be shown to a user 14 watching the virtual viewpoint video 78 via the first advertising area 79.

[0234] Furthermore, in the image processing device 10 according to the seventh modification, second timing-related information 88G is included in the second advertising area-related information 88 as information related to the timing at which the first advertising area 79 is included in the virtual viewpoint video 78 (see FIG. 32). Furthermore, third timing-related information 84H is included in the advertising video-related information 84 as information related to the timing to be compared with the second timing-related information 88G when determining the degree of coincidence between the second advertising area-related information 88 and the advertising video-related information 84 (see FIG. 33). Therefore, according to this configuration, an advertising video 60B for the second advertising area (for example, an advertising video 60 associated with advertising video related information 84 including third timing related information 84H that coincides with the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 and the timing at which the advertising video 60 is included in the virtual viewpoint video 78) selected from a plurality of advertising videos 60 based on the timing at which the second advertising area 86 is included in the virtual viewpoint video 78 and the timing at which the advertising video 60 is included in the virtual viewpoint video 78 can be shown to a user 14 watching the virtual viewpoint video 78 via the second advertising area 86.

[0235] [Eighth Modification] Weights may be assigned to the second content-related information 84A, the third advertising effect-related information 84B, the third size-related information 84C, the third viewpoint information 84D, the third display time-related information 84E, the total playback time information 84F, the third movement speed-related information 84G, and the third timing-related information 84H included in the advertising video-related information 84 by the party providing the advertising video 60 to the user 14, the producer of the advertising video 60, and / or the producer of the image processing system 2. In this case, the degree of match between the various pieces of information included in the first advertising area-related information 82 and the various pieces of information included in the advertising video-related information 84 is calculated by multiplying the weights. For example, if a weight of "1" is assigned to one of the multiple pieces of information included in the advertising video-related information 84 and a weight of "0" is assigned to the remaining pieces of information, only the degree of match related to the information weighted "1" is calculated.

[0236] Similarly, weights may be assigned to the various pieces of information contained in the first advertising area related information 82 and / or the various pieces of information contained in the second advertising area related information 88 by the producer of the advertising video 60 and / or the producer of the image processing system 2, etc.

[0237] [Ninth Variation] 34, the display mode of the advertising video 60 may be changed depending on the viewpoint VP. In this case, for example, one piece of advertising video related information 84 and a plurality of advertising videos 60 with different orientations are stored in association with each other in the storage 30. The plurality of advertising videos 60 with different orientations refers to a plurality of patterns of video showing the appearance of the advertisement shown by the advertising video 60 when viewed from a plurality of orientations.

[0238] When the advertising video 60A for the first advertising area is displayed in the first advertising area 79, the screen data generation unit 28D acquires, from among the multiple advertising videos 60 associated with advertising video related information 84 that has the highest degree of match with the first advertising area related information 82, that is, from the multiple advertising videos 60 with different orientations, the advertising video 60 with the orientation that has the highest degree of match with the orientation (e.g., line of sight direction) of the viewpoint VP identified from the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B, as the advertising video 60A for the first advertising area.

[0239] Then, the screen data generation unit 28D generates virtual viewpoint video screen data 80 including a first advertising area 79 in which the advertising video for the first advertising area 60A is displayed. Every time the content of the viewpoint information 74 is updated, the screen data generation unit 28D performs a similar process, thereby generating virtual viewpoint video screen data 80 showing the virtual viewpoint video screen 68 in which the orientation of the advertising video for the first advertising area 60A has been changed according to the viewpoint information 74. This makes it possible to show the advertising video for the first advertising area 60A to the user 14 watching the virtual viewpoint video 78 via the first advertising area 79 in an appropriate orientation according to the viewpoint VP, compared to when the orientation of the advertising video for the first advertising area 60A is always constant regardless of the viewpoint VP (when there is only one pattern).

[0240] In addition, when the advertising video 60B for the second advertising area is displayed in the second advertising area 86, the screen data generation unit 28D acquires, from among the multiple advertising videos 60 associated with advertising video related information 84 that has the highest degree of match with the second advertising area related information 88, that is, from the multiple advertising videos 60 with different orientations, the advertising video 60 with the orientation that has the highest degree of match with the orientation (e.g., line of sight direction) of the viewpoint VP identified from the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B, as the advertising video 60B for the second advertising area.

[0241] Then, the screen data generation unit 28D generates virtual viewpoint video screen data 80 including a second advertising area 86 in which the advertising video for the second advertising area 60B is displayed. Every time the content of the viewpoint information 74 is updated, the screen data generation unit 28D performs a similar process, thereby generating virtual viewpoint video screen data 80 that shows the virtual viewpoint video screen 68 in which the orientation of the advertising video for the second advertising area 60B has been changed in accordance with the viewpoint information 74. This makes it possible to show the advertising video for the second advertising area 60B to the user 14 watching the virtual viewpoint video 78 via the second advertising area 86 in an appropriate orientation according to the viewpoint VP, compared to when the orientation of the advertising video for the second advertising area 60B is always constant regardless of the viewpoint VP (when there is only one pattern).

[0242] In the example shown in Figure 34, orientation is given as an example of the display mode of the advertising video 60, but the technology of the present disclosure is not limited to this, and the brightness and / or contrast of the advertising video 60 may be applied as the display mode of the advertising video 60 in addition to or instead of the orientation of the advertising video 60.

[0243] [Tenth Modification] The first advertising area 79 and the second advertising area 86 may be displayed on the virtual viewpoint video screen 68 at different times. In this case, for example, as shown in Fig. 35, the screen data generation unit 28D generates virtual viewpoint video screen data 80 as data for displaying the first advertising area 79 including the advertising video 60A for the first advertising area on the virtual viewpoint video screen 68 before the second advertising area 86 including the advertising video 60B for the second advertising area. Conversely, the screen data generation unit 28D may generate virtual viewpoint video screen data 80 as data for displaying the second advertising area 86 including the advertising video 60B for the second advertising area on the virtual viewpoint video screen 68 before the first advertising area 79 including the advertising video 60A for the first advertising area.

[0244] The display order of the first advertising area 79 including the advertising video for the first advertising area 60A and the second advertising area 86 including the advertising video for the second advertising area 60B may be determined, for example, according to various information included in the first advertising area related information 82 and / or various information included in the second advertising area related information 88, or may be determined according to various information included in the advertising video related information 84 such as the rank identifier 90, or may be determined according to instructions received by the receiving device 50, etc.

[0245] In this way, by differentiating the timing at which the first advertising area 79 and the second advertising area 86 are displayed within the virtual viewpoint video screen 68, it is possible to differentiate the advertising effectiveness of the advertising video 60A for the first advertising area from the advertising effectiveness of the advertising video 60B for the second advertising area.

[0246] Furthermore, in each of the above embodiments and variant examples, an example in which the screen generation processing is performed by the computer 22 of the image processing device 10 has been described, but the technology of the present disclosure is not limited to this, and the screen generation processing may be performed by the computer 40 of the user device 12, or distributed processing may be performed by the computer 22 of the image processing device 10 and the computer 40 of the user device 12.

[0247] Furthermore, while the above embodiments and modifications have exemplified the computer 22, the technology of the present disclosure is not limited to this. For example, a device including an ASIC, FPGA, and / or PLD may be applied instead of the computer 22. Furthermore, a combination of hardware and software configurations may be used instead of the computer 22. The same applies to the computer 40 of the user device 12.

[0248] Furthermore, in the above-described embodiments and modifications, the screen generation processing program 38 is stored in the storage 30, but the technology of the present disclosure is not limited to this, and as an example, the screen generation processing program 38 may be stored in any portable storage medium 200 such as an SSD or USB memory, which is a non-transitory storage medium, as shown in Fig. 36. In this case, the screen generation processing program 38 stored in the storage medium 200 is installed in the computer 22, and the processor 28 executes the screen generation processing in accordance with the screen generation processing program 38.

[0249] Furthermore, the screen generation processing program 38 may be stored in the memory of another computer or server device connected to the computer 22 via a communication network (not shown), and the screen generation processing program 38 may be downloaded to the image processing device 10 in response to a request from the image processing device 10. In this case, the screen generation processing is executed by the processor 28 of the computer 22 in accordance with the downloaded screen generation processing program 38.

[0250] Furthermore, in the above example, processor 28 is illustrated, but instead of processor 28, or together with processor 28, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0251] The hardware resources that execute the screen generation process can be various processors, as listed below. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource that executes the screen generation process according to software, i.e., a program, as described above. Other processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically to execute dedicated processes. Each processor has built-in or connected memory, and executes the screen generation process by using the memory.

[0252] The hardware resource that executes the screen generation process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the screen generation process may be a single processor.

[0253] Examples of a system configured with a single processor include: first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes screen generation processing; second, a system in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute screen generation processing, on a single IC chip, as typified by SoCs; thus, the screen generation processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0254] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0255] Furthermore, the above-described screen generation process is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0256] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0257] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0258] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; a memory connected to or embedded in the processor; The processor: acquiring a virtual viewpoint image generated based on a plurality of captured images; outputting first data for displaying a first specific image selected from the plurality of specific images in the first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images not included in the plurality of captured images; Image processing device.

2. The first information includes first content-related information related to the content of the virtual viewpoint image. The image processing device according to claim 1 .

3. the specific image-related information includes second content-related information related to the content of the specific image; The first specific image is a specific image among the plurality of specific images that is related to the specific image-related information that includes the second content-related information corresponding to the first content-related information. The image processing device according to claim 2 .

4. The first information includes first advertising effect related information regarding advertising effect. The image processing device according to any one of claims 1 to 3.

5. The first information includes first size-related information regarding a size at which the first specific image is displayed in the first area. The image processing device according to any one of claims 1 to 4.

6. The first information includes first viewpoint information required to generate the virtual viewpoint image. The image processing device according to any one of claims 1 to 5.

7. The first viewpoint information includes information about a first viewpoint path. The image processing device according to claim 6 .

8. The first information includes first display time related information related to a time when the first area is displayed. The image processing device according to any one of claims 1 to 7.

9. The first display time related information is information related to the time during which the first area is continuously displayed. The image processing device according to claim 8 .

10. the specific image is a moving image, the specific image-related information includes a total playback time of the video; The processor generates the first data based on the first display time related information and the total playback time.

10. The image processing device according to claim 8 or claim 9.

11. the specific image is a moving image, the specific image-related information includes a total playback time of the video; The processor selects the first specific image based on the first display time related information and the total playback time. The image processing device according to any one of claims 8 to 10.

12. the virtual viewpoint image is a moving image, The first information includes first timing-related information related to the timing at which the first region is included in the virtual viewpoint image. The image processing device according to any one of claims 1 to 11.

13. The first information includes first moving speed related information related to the moving speed of the first viewpoint required to generate the virtual viewpoint image. The image processing device according to any one of claims 1 to 12.

14. The first information is changed according to at least one of a viewpoint position, a line of sight direction, and an angle of view required for generating the virtual viewpoint image. The image processing device according to any one of claims 1 to 13.

15. The processor further comprises: outputting second data for displaying a second specific image selected from the plurality of specific images in the second area based on second information associated with the second area related to the virtual viewpoint image and the specific image related information; The image processing device according to any one of claims 1 to 14.

16. The second information includes second advertising effectiveness-related information regarding advertising effectiveness. The image processing device according to claim 15.

17. The second information includes second size-related information relating to a size at which the second specific image is displayed in the second area.

17. The image processing device according to claim 15 or 16.

18. The second information includes second viewpoint information required to generate the virtual viewpoint image.

18. The image processing device according to claim 15.

19. The second viewpoint information includes information about a second viewpoint path. The image processing device according to claim 18.

20. The second information includes second display time related information related to a time when the second area is displayed.

20. The image processing device according to claim 15,

21. The second display time related information is information related to the time during which the second area is continuously displayed. The image processing device according to claim 20.

22. the specific image is a moving image, the specific image-related information includes a total playback time of the video; The processor generates the second data based on the second display time related information and the total playback time. The image processing device according to claim 20.

23. the specific image is a moving image, the specific image-related information includes a total playback time of the video; The processor selects the second specific image based on the second display time related information and the total playback time.

23. The image processing device according to any one of claims 20 to 22.

24. the virtual viewpoint image is a moving image, The second information includes second timing-related information related to the timing at which the second area is included in the virtual viewpoint image.

24. The image processing device according to claim 15.

25. The second information includes second movement speed related information related to the movement speed of the second viewpoint required to generate the virtual viewpoint image.

25. The image processing device according to any one of claims 15 to 24.

26. The second information is changed according to at least one of a viewpoint position, a line of sight direction, and an angle of view required for generating the virtual viewpoint image.

26. The image processing device according to any one of claims 15 to 25.

27. The specific image related information includes billing information of the party that provides the specific image.

27. The image processing device according to any one of claims 1 to 26.

28. The display mode of the specific image is changed depending on the viewpoint used to generate the virtual viewpoint image.

28. The image processing device according to any one of claims 1 to 27.

29. a processor; a memory connected to or embedded in the processor; The processor: acquiring a virtual viewpoint image generated based on a plurality of captured images; outputting data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with a plurality of regions related to the virtual viewpoint image and specific image-related information associated with a specific image not included in the plurality of captured images; Image processing device.

30. The timing at which the plurality of regions are displayed is different 30. The image processing device according to claim 29.

31. Obtaining a virtual viewpoint image generated based on a plurality of captured images; and outputting first data for displaying a first specific image selected from the plurality of specific images in the first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images not included in the plurality of captured images; An image processing method comprising:

32. Obtaining a virtual viewpoint image generated based on a plurality of captured images; and outputting data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with the plurality of regions related to the virtual viewpoint image and specific image-related information associated with a specific image not included in the plurality of captured images; An image processing method comprising:

33. On the computer, Obtaining a virtual viewpoint image generated based on a plurality of captured images; and outputting first data for displaying a first specific image selected from the plurality of specific images in the first area based on first information associated with a first area related to the virtual viewpoint image and specific image-related information related to a plurality of specific images not included in the plurality of captured images; A program for executing a process including:

34. On the computer, Obtaining a virtual viewpoint image generated based on a plurality of captured images; and outputting data for displaying a specific image in at least one of the plurality of regions based on region-related information associated with the plurality of regions related to the virtual viewpoint image and specific image-related information associated with a specific image not included in the plurality of captured images; A program for executing a process including:

Images