Image processing device, image processing method, and program

Abstract

This image processing device comprises a processor and a memory connected to or incorporated in the processor. The processor acquires, on the basis of a plurality of captured images and viewpoint information, a virtual viewpoint image indicating the appearance of a subject as the subject is observed from the viewpoint identified by the viewpoint information, and outputs, on the basis of the viewpoint information, data for causing a display to show a specific image created in a process different from that for the virtual viewpoint image and the virtual viewpoint image.

Description

Technical Field

[0001] The technology of the present disclosure relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Japanese Patent Application Laid-Open No. 2018-142164 discloses an image processing apparatus included in a system in which a virtual viewpoint image in which a virtual object not included in a plurality of captured images is inserted is generated based on a plurality of captured images obtained by capturing a subject from different directions by a plurality of cameras and information corresponding to the designation of a virtual viewpoint. The image processing apparatus includes an acquisition unit that acquires viewpoint information for specifying a movement path of a virtual viewpoint related to the generation of the virtual viewpoint image, and a display area in the virtual viewpoint image corresponding to the virtual viewpoint at a first time point on the movement path specified from the viewpoint information acquired by the acquisition unit, the display area being an area for displaying a virtual object, and a control unit that determines the display area based on the virtual viewpoint at the first time point and a virtual viewpoint on the movement path that is a virtual viewpoint at a second time point different from the first time point. Japanese Patent Application Laid-Open No. 2018-142164 discloses that the virtual object is an object for displaying an advertisement in the virtual viewpoint image.

[0003] Japanese Patent Application Laid-Open No. 2020-101847 discloses an image file generation apparatus that generates an image file for generating a virtual viewpoint image, the image file generation apparatus including a material information acquisition unit that acquires material information used for generating the virtual viewpoint image, an additional information acquisition unit that acquires additional information to be displayed in the virtual viewpoint image, and an image file generation unit that generates an image file including the material information and the additional information.

[0004] Japanese Patent Publication No. 2012-048639 discloses a free-viewpoint video generation device comprising: a holding means for holding subject and background data for generating a free-viewpoint video; a generating means for generating a video viewed from a received viewpoint using the data held in the holding means; a determining means for determining an area for inserting a desired advertisement from the background area of ​​the video generated by the generating means; and a combining means for generating a video in which the desired advertisement is pasted onto the area for inserting the advertisement in the video generated by the generating means. [Overview of the project]

[0005] One embodiment of the technology of this disclosure provides an image processing device, an image processing method, and a program that can show a specific image to a viewer of a virtual viewpoint image. [Means for solving the problem]

[0006] A first aspect of the technology of this disclosure is an image processing device comprising a processor and memory connected to or built into the processor, wherein the processor acquires a virtual viewpoint image showing the appearance of a subject when the subject is observed from a viewpoint identified by the viewpoint information, based on a plurality of captured images and viewpoint information, and outputs data for displaying a specific image created in a process different from the virtual viewpoint image and the virtual viewpoint image on a display based on the viewpoint information.

[0007] A second aspect of the technology of this disclosure is an image processing device comprising a processor and memory connected to or built into the processor, wherein the processor acquires a virtual viewpoint image showing the appearance of a subject when the subject is observed from a viewpoint specified by the viewpoint information, based on a plurality of captured images and viewpoint information, and outputs data for displaying a specific image created without using a plurality of captured images and the virtual viewpoint image on a display based on the viewpoint information.

[0008] A third aspect of the technology of this disclosure is an image processing device comprising a processor and memory connected to or built into the processor, wherein the processor acquires a virtual viewpoint image showing the appearance of a subject when the subject is observed from a viewpoint specified by the viewpoint information, based on a plurality of captured images and viewpoint information, outputs data for displaying the virtual viewpoint image on a display, and outputs data for displaying a specific image on the display at a timing determined according to the viewpoint information.

[0009] A fourth aspect of the technology of this disclosure is that the viewpoint information includes a time parameter relating to time, and the data causes a virtual viewpoint image to be displayed on a display. for An image processing device relating to any one of the first to third embodiments, which includes first data for displaying a specific image on a display according to a time parameter.

[0010] A fifth aspect of the technology of this disclosure is an image processing device relating to any one of the first to fourth aspects, wherein the viewpoint information includes setting completion information indicating that the setting of viewpoint information has been completed, and the data is second data for displaying a virtual viewpoint image on a display, and the second data includes, in accordance with the setting completion information, second data for displaying a specific image on the display between the completion of the setting of viewpoint information and the display of the virtual viewpoint image on the display.

[0011] A sixth aspect of the technology of this disclosure is an image processing device relating to any one of the first to fifth aspects, wherein the data is third data for displaying a virtual viewpoint image on a display, and includes third data for displaying a specific image on the display in accordance with the timing at which the continuity of viewpoint information is interrupted.

[0012] A seventh aspect of the technology of this disclosure is an image processing device relating to any one of the first to sixth aspects, wherein the viewpoint information includes viewpoint path information indicating a viewpoint path for observing a subject, and the data is fourth data for displaying a virtual viewpoint image on a display, and the fourth data for displaying a specific image on a display at intervals divided by the viewpoint path indicated by the viewpoint path information.

[0013] An eighth aspect of the technology of this disclosure is an image processing device relating to any one of the first to seventh aspects, wherein the viewpoint information includes time information indicating the time required for a first viewpoint observing a subject to move from a first position to a second position different from the first position, and the data includes fifth data for displaying a virtual viewpoint image on a display, and the fifth data for displaying a specific image on the display at intervals divided by the time required indicated by the time information.

[0014] A ninth aspect of the technology of this disclosure is an image processing device relating to any one of the first to eighth aspects, wherein the viewpoint information includes the position of a second viewpoint for observing a subject and elapsed time information indicating the elapsed time corresponding to the position of the second viewpoint, and the data includes sixth data for displaying a virtual viewpoint image on a display, and sixth data for displaying a specific image on the display at a timing determined according to the relationship between the elapsed time indicated by the elapsed time information and the position of the second viewpoint.

[0015] A tenth aspect of the technology of this disclosure is an image processing device relating to any one of the first to ninth aspects, wherein the viewpoint information includes movement speed information that can identify the movement speed of the position of a third viewpoint observing a subject, and the data includes seventh data for displaying a virtual viewpoint image on a display, and the seventh data for displaying a specific image on the display at the timing when the movement speed identified from the movement speed information falls below a threshold.

[0016] An eleventh aspect of the technology of this disclosure is an image processing device relating to any one of the first to tenth aspects, wherein the viewpoint information includes angle of view information relating to the angle of view for observing a subject, and the data includes eighth data for displaying a virtual viewpoint image on a display, and eighth data for displaying a specific image on the display at a timing determined according to the angle of view information.

[0017] A twelfth aspect of the technology of this disclosure is an image processing device relating to any one of the first to eleventh aspects, wherein the data is ninth data for displaying a virtual viewpoint image on a display, and includes ninth data for displaying a specific image on the display at the timing when the display of the virtual viewpoint image by the display begins, or ninth data for displaying a virtual viewpoint image on a display, and includes ninth data for displaying a virtual viewpoint image on the display at the timing when the display of the specific image by the display ends.

[0018] A thirteenth aspect of the technology of this disclosure is an image processing apparatus according to a twelfth aspect, wherein the processor outputs tenth data for displaying a reception screen that accepts viewpoint information on a display, and outputs data including ninth data on the condition that the reception of viewpoint information by the reception screen has been completed.

[0019] A fourteenth aspect of the technology of this disclosure is an image processing device according to any one of the first to thirteenth aspects, wherein the processor further acquires gaze point information that can identify the position of the gaze point, and controls the display timing of a specific image according to the fluctuations of the gaze point identified from the gaze point information.

[0020] A 15th aspect of the technology of this disclosure is an image processing method that includes acquiring a virtual viewpoint image showing the appearance of a subject when it is observed from a viewpoint identified by viewpoint information, based on a plurality of captured images and viewpoint information generated by capturing a subject by a plurality of imaging devices, and outputting a specific image created in a process different from the virtual viewpoint image, and data for displaying the virtual viewpoint image on a display based on the viewpoint information.

[0021] A sixteenth aspect of the technology of this disclosure is an image processing method that includes acquiring a virtual viewpoint image showing the appearance of a subject when the subject is observed from a viewpoint identified by viewpoint information, based on a plurality of captured images and viewpoint information, and outputting data based on viewpoint information for displaying a specific image created without using a plurality of captured images and the virtual viewpoint image on a display based on viewpoint information.

[0022] A 17th aspect of the technology of this disclosure is an image processing method that includes acquiring a virtual viewpoint image showing the appearance of a subject when it is observed from a viewpoint identified by viewpoint information, based on a plurality of captured images and viewpoint information generated by capturing a subject by a plurality of imaging devices, outputting data for displaying the virtual viewpoint image on a display, and outputting data for displaying a specific image on a display at a timing determined according to the viewpoint information.

[0023] An eighteenth aspect of the technology of this disclosure is a program that causes a computer to perform a process including obtaining a virtual viewpoint image showing the appearance of a subject when it is observed from a viewpoint identified by viewpoint information, based on a plurality of captured images and viewpoint information generated when the subject is captured by a plurality of imaging devices, and outputting a specific image created in a process different from the virtual viewpoint image, and data for displaying the virtual viewpoint image on a display based on the viewpoint information.

[0024] A 19th aspect of the technology of the present disclosure is a program for causing a computer to execute a process including: obtaining a virtual viewpoint image showing an aspect of a subject when the subject is observed from a viewpoint specified by viewpoint information based on a plurality of captured images and the viewpoint information; and outputting data for causing a display to display a specific image created without using the plurality of captured images and the virtual viewpoint image based on the viewpoint information.

[0025] A 20th aspect of the technology of the present disclosure is a program for causing a computer to execute a process including: obtaining a virtual viewpoint image showing an aspect of a subject when the subject is observed from a viewpoint specified by viewpoint information based on a plurality of captured images generated by capturing the subject with a plurality of imaging devices and the viewpoint information; outputting data for causing a display to display the virtual viewpoint image; and outputting data for causing a display to display a specific image at a timing determined according to the viewpoint information.

Brief Description of the Drawings

[0026] [Figure 1] It is a conceptual diagram showing an example of the configuration of an image processing system. [Figure 2] It is a block diagram showing an example of the hardware configuration of the electrical system of a user device. [Figure 3] It is a block diagram showing an example of the main functions of the CPU of an image processing device. [Figure 4] It is a conceptual diagram showing an example of the processing contents of a reception screen generation unit and an output unit, and an example of the display contents of a display of a user device. [Figure 5] It is a screen diagram showing an example of the display mode of a reception screen when the operation mode of a user device is a viewpoint setting mode. [Figure 6] It is a screen diagram showing an example of the display mode of a reception screen when the operation mode of a user device is a fixation point setting mode. [Figure 7]This block diagram shows an example of the content of viewpoint information and an example of how viewpoint information is transmitted from a user device to an image processing device. [Figure 8] This is a conceptual diagram illustrating an example of the processing content of the virtual viewpoint image generation unit. [Figure 9] This is a conceptual diagram showing an example of the processing content of the screen data generation and output units. [Figure 10] This is a screen diagram showing an example of a configuration in which a virtual viewpoint video containing advertisements and a reception screen are displayed on the user device's display. [Figure 11] This is a conceptual diagram showing an example of the content of the first screen data generated by the screen data generation unit. [Figure 12] This is a conceptual diagram showing an example of the content of the second screen data generated by the screen data generation unit. [Figure 13] This is a conceptual diagram illustrating an example of how a third screen data is generated by the screen data generation unit when the continuity of viewpoint information is interrupted. [Figure 14] This is a conceptual diagram showing an example of the content of the third screen data generated by the screen data generation unit. [Figure 15] This is a conceptual diagram showing an example of the content of the fourth screen data generated by the screen data generation unit. [Figure 16] This is a conceptual diagram showing an example of the content of the fifth screen data generated by the screen data generation unit. [Figure 17] This is a conceptual diagram showing an example of the content of the sixth screen data generated by the screen data generation unit. [Figure 18] This is a conceptual diagram showing an example of the content of the seventh screen data generated by the screen data generation unit. [Figure 19] This is a conceptual diagram showing an example of the content of the eighth screen data generated by the screen data generation unit. [Figure 20] This is a flowchart illustrating an example of the screen generation process. [Figure 21] This is a conceptual diagram showing an example of the content of the ninth screen data generated by the screen data generation unit. [Figure 22]This is a conceptual diagram showing an example of the content of screen data generated by the screen data generation unit. [Figure 23] This is a conceptual diagram illustrating an example of how an image processing system handles situations where the point of focus changes abruptly. [Figure 24] This is a conceptual diagram showing a modified configuration of an image processing system. [Figure 25] This is a conceptual diagram illustrating an example of how a screen generation processing program stored on a storage medium is installed on the computer of an image processing device. [Modes for carrying out the invention]

[0027] An example of an embodiment of the image processing apparatus, image processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0028] First, let's explain the terminology used in the following explanation.

[0029] CPU stands for "Central Processing Unit". GPU stands for "Graphics Processing Unit". TPU stands for "Tensor Processing Unit". NVM stands for "Non-Volatile Memory". RAM stands for "Random Access Memory". SSD stands for "Solid State Drive". HDD stands for "Hard Disk Drive". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". I / F stands for "Interface". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device." EL stands for "Electro-Luminescence." LAN stands for "Local Area Network." USB stands for "Universal Serial Bus." HMD LTE stands for "Head Mounted Display". 5G stands for "5th generation (wireless technology for digital cellular networks)". TDM stands for "Time-Division Multiplexing".

[0030] As an example, as shown in Figure 1, the image processing system 2 includes an image processing device 10 and a user device 12.

[0031] In this embodiment, a server is used as an example of the image processing device 10. The server is implemented by, for example, a mainframe. However, this is merely an example, and the server may be implemented by network computing such as cloud computing, fog computing, edge computing, or grid computing. Furthermore, the image processing device 10 may be a personal computer, multiple personal computers, multiple servers, or a device that combines a personal computer and a server.

[0032] Furthermore, in this embodiment, a smartphone is used as an example of the user device 12. However, the smartphone is merely an example; for example, it could be a personal computer, a tablet device, or a portable multi-functional device such as a head-mounted display (hereinafter referred to as "HMD").

[0033] Furthermore, in this embodiment, the image processing device 10 and the user device 12 are connected communicably via, for example, a base station (not shown). The communication standards used by the base station include wireless communication standards such as 5G and / or LTE, wireless communication standards such as WiFi (802.11) and / or Bluetooth®, and wired communication standards such as TDM and / or Ethernet®.

[0034] The image processing device 10 acquires an image and transmits the acquired image to the user device 12. Here, "image" refers to, for example, the captured image 64 obtained by imaging (see Figures 3, 4, and 8, etc.), and images generated based on the captured image 64 (see Figures 3, 4, and 8, etc.). An example of an image generated based on the captured image (see Figures 3, 4, and 8, etc.) is the virtual viewpoint image 76 (see Figure 8, etc.).

[0035] The user device 12 is used by user 14. The user device 12 is equipped with a touch panel display 16. The touch panel display 16 is implemented by a display 18 and a touch panel 20. An example of the display 18 is an EL display (e.g., an organic EL display or an inorganic EL display). Note that it is not limited to EL displays, but also includes liquid crystal displays. vinegar Other types of displays, such as those for playing games, are also acceptable.

[0036] The touch panel display 16 is formed by superimposing the touch panel 20 onto the display area of ​​the display 18, or by using an in-cell type in which the touch panel function is built into the display 18. Note that the in-cell type is merely one example, and it may also be an out-cell type or an on-cell type.

[0037] The user device 12 performs processing in accordance with instructions received from the user 14 via the touch panel 20 or the like. For example, the user device 12 exchanges various types of information with the image processing device 10 in accordance with instructions received from the user 14 via the touch panel 20 or the like.

[0038] User device 12 receives an image transmitted from image processing device 10 and displays the received image on display 18. User 14 views the image displayed on display 18.

[0039] The image processing device 10 includes a computer 22, a transceiver 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure and includes a CPU 28, an NVM 30, and RAM 32. The image processing device 10 includes a bus 34, and the CPU 28, NVM 30, and RAM 32 are connected via the bus 34. In the example shown in Figure 1, for illustrative purposes, one bus is shown as bus 34, but there may be multiple buses. Furthermore, bus 34 may include a serial bus, or a parallel bus composed of a data bus, an address bus, and a control bus, etc.

[0040] The CPU 28 is an example of a "processor" related to the technology of this disclosure. The CPU 28 controls the entire image processing device 10. The NVM 30 stores various parameters and various programs, etc. Examples of the NVM 30 include EEPROM, SSD, and / or HDD. The RAM 32 is an example of a "memory" related to the technology of this disclosure. Various information is temporarily stored in the RAM 32. The RAM 32 is used as work memory by the CPU 28.

[0041] The transceiver 24 is connected to the bus 34. The transceiver 24 is a device that includes a communication processor (not shown) and an antenna, etc., and under the control of the CPU 28, it transmits and receives various information with the user device 12 via a base station (not shown). In other words, the CPU 28 exchanges various information with the user device 12 via the transceiver 24.

[0042] The communication interface 26 is implemented, for example, by a device having an FPGA. The communication interface 26 is connected to multiple imaging devices 36 via LAN cables (not shown). The imaging devices 36 are imaging devices having a CMOS image sensor and are equipped with optical zoom and / or digital zoom functions. Note that other types of image sensors, such as CCD image sensors, may be used instead of CMOS image sensors.

[0043] For example, multiple imaging devices 36 are installed in a soccer stadium (not shown) and capture images of subjects inside the soccer stadium. The captured images 64 (see Figures 3, 4, and 8, etc.) obtained by the imaging devices 36 capturing images of subjects are used, for example, to generate virtual viewpoint images 76 (see Figure 8, etc.). Therefore, each of the multiple imaging devices 36 is installed in a different location within the soccer stadium, that is, in a location where multiple captured images 64 (see Figures 3, 4, and 8, etc.) capable of generating virtual viewpoint images 76 (see Figure 8, etc.) can be obtained. The soccer stadium is a three-dimensional area including a soccer field (not shown) and spectator seats (not shown) constructed surrounding the soccer field, and is the object of observation for user 14. The observer, i.e., user 14, can observe the inside of the soccer stadium from the spectator seats or from a location outside the soccer stadium through images displayed on the display 18 of the user device 12.

[0044] In this document, a soccer stadium is given as an example of a location where multiple imaging devices 36 are installed. However, the technology of this disclosure is not limited to this, and the location where multiple imaging devices 36 are installed can be any location where multiple imaging devices 36 can be installed, such as a baseball field, rugby field, curling rink, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater.

[0045] The communication interface 26 is connected to the bus 34 and is responsible for exchanging various types of information between the CPU 28 and the multiple imaging devices 36. For example, the communication interface 26 controls the multiple imaging devices 36 according to the requests of the CPU 28. The communication interface 26 outputs the captured images 64 (see Figures 3, 4, and 8, etc.) obtained by each of the multiple imaging devices 36 to the CPU 28. Here, the communication interface 26 is exemplified as a wired communication interface, but it may also be a wireless communication interface such as a high-speed wireless LAN.

[0046] The NVM 30 stores a screen generation processing program 38. The screen generation processing program 38 is an example of a "program" related to the technology of this disclosure. The CPU 28 reads the screen generation processing program 38 from the NVM 30 and performs screen generation processing (see Figure 20) by executing the screen generation processing program 38 on the RAM 32.

[0047] As an example, as shown in Figure 2, the user device 12 includes a display 18, a computer 40, an imaging device 42, a transceiver 44, a speaker 46, a microphone 48, and a reception device 50. The computer 40 includes a CPU 52, an NVM 54, and RAM 56. The user device 12 is equipped with a bus 58, and the CPU 52, NVM 54, and RAM 56 are connected via the bus 58.

[0048] In the example shown in Figure 2, for illustrative purposes, a single bus is depicted as bus 58, but there may be multiple buses. Furthermore, bus 58 may include a serial bus, or a parallel bus composed of a data bus, address bus, and control bus, etc.

[0049] The CPU 52 controls the entire user device 12. The NVM 54 stores various parameters and programs. An example of the NVM 54 is an EEPROM. Various information is temporarily stored in the RAM 56. The RAM 56 is used as work memory by the CPU 52. The CPU 52 reads various programs from the NVM 54 and executes them on the RAM 56, performing processing according to the programs.

[0050] The imaging device 42 is an imaging device having a CMOS image sensor and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The imaging device 42 is connected to the bus 58, and the CPU 52 controls the imaging device 42. The captured images obtained by the imaging device 42 are acquired by the CPU 52 via the bus 58.

[0051] The transceiver 44 is connected to the bus 58. The transceiver 44 is a device that includes a communication processor (not shown) and an antenna, and under the control of the CPU 52, it transmits and receives various information to and from the image processing device 10 via a base station (not shown). In other words, the CPU 52 exchanges various information with the image processing device 10 via the transceiver 44.

[0052] Speaker 46 converts electrical signals into sound. Speaker 46 is connected to bus 58. Speaker 46 receives electrical signals output from CPU 52 via bus 58, converts the received electrical signals into sound, and outputs the resulting sound to the outside of user device 12.

[0053] Microphone 48 converts the collected sound into an electrical signal. Microphone 48 is connected to bus 58. The electrical signal obtained by converting the sound collected by microphone 48 is acquired by CPU 52 via bus 58.

[0054] The receiving device 50 receives instructions from the user 14, etc. Examples of the receiving device 50 include a touch panel 20 and hard keys (not shown). The receiving device 50 is connected to the bus 58, and the instructions received by the receiving device 50 are acquired by the CPU 52.

[0055] As an example, as shown in Figure 3, the NVM 30 of the image processing device 10 stores multiple advertising videos 62. The advertising videos 62 are videos created through a different process than the virtual viewpoint image 76 (see Figure 8, etc.). The multiple advertising videos 62 stored in the NVM 30 are selectively read from the NVM 30 by the CPU 28 and used in the screen generation process. In the example shown in Figure 3, multiple advertising videos 62 are stored in the NVM 30, but this is just one example, and a single advertising video 62 may be stored in the NVM 30.

[0056] The advertising video 62 is a video that represents an advertisement (for example, a moving image obtained by capturing a subject in a real-world area with a camera, and / or an animation, etc.). A video that represents an advertisement refers to, for example, a moving image in which multiple frames created as images for advertising are arranged in chronological order. An example of an advertising video 62 is a video provided by a sponsor or the like that is supporting the construction of a system (for example, the image processing system 2 shown in Figure 1) that allows the user 14 to view various videos, including virtual viewpoint video 78 (see Figure 8).

[0057] Here, a moving image is used as an example of the advertising video 62, but the technology of this disclosure is not limited to this, and may also be a single-frame image for advertising or an image used for purposes other than advertising. The advertising video 62 is merely an example, and may be other types of moving images or still images. The advertising video 62 is an example of a "specific image" related to the technology of this disclosure.

[0058] The CPU 28 reads the screen generation processing program 38 from the NVM 30 and executes the screen generation processing program 38 on the RAM 32, thereby operating as the reception screen generation unit 28A, the viewpoint information acquisition unit 28B, the virtual viewpoint image generation unit 28C, the screen data generation unit 28D, and the output unit 28E.

[0059] As an example, as shown in Figure 4, the user device 12's display 18 shows the reception screen 66 and the virtual viewpoint video screen 68. In the example shown in Figure 4, the reception screen 66 and the virtual viewpoint video screen 68 are displayed side by side on the display 18. This is merely an example, and the reception screen 66 and the virtual viewpoint video screen 68 may be switched and displayed according to instructions given by the user 14 to the touch panel 20, or the reception screen 66 and the virtual viewpoint video screen 68 may be displayed separately on different display devices.

[0060] Furthermore, in this embodiment, the reception screen 66 is displayed on the display 18 of the user device 12, but the technology of this disclosure is not limited thereto. For example, the reception screen 66 may be displayed on a display connected to a device (e.g., a personal computer) used by a person who creates or edits virtual viewpoint video 78 (see Figure 8).

[0061] The user device 12 communicates with the image processing device 10 to acquire a virtual viewpoint image 78 (see Figure 8) from the image processing device 10. The virtual viewpoint image screen 68 on the display 18 displays the virtual viewpoint image 78 (see Figure 8) acquired by the user device 12 from the image processing device 10. In the example shown in Figure 4, the virtual viewpoint image 78 is not displayed on the virtual viewpoint image screen 68.

[0062] The user device 12 communicates with the image processing device 10 and obtains reception screen data 70 indicating the reception screen 66 from the image processing device 10. The display 18 displays the reception screen 66 indicated by the reception screen data 70 obtained from the image processing device 10 by the user device 12.

[0063] The reception screen 66 includes an overhead view screen 66A, a guidance message display area 66B, a confirmation key 66C, and a cancel key 66D, and displays various information necessary for generating the virtual viewpoint image 78 (see Figure 8). The user 14 gives instructions to the user device 12 by referring to the reception screen 66. Instructions from the user 14 are received, for example, by the touch panel 20.

[0064] The overhead view screen 66A displays the overhead view 72. The overhead view 72 is a moving image showing what the inside of a soccer stadium looks like when viewed from above, and is generated based on multiple captured images 64 obtained by capturing images with at least one of the multiple imaging devices 36. An example of the overhead view 72 is a live broadcast video.

[0065] The guidance message display area 66B displays various messages indicating the content of the operations required of the user 14. The operations required of the user 14 refer to, for example, the operations necessary for generating the virtual viewpoint image 78 (see Figure 8) (for example, operations to set the viewpoint and operations to set the point of focus).

[0066] The content displayed in the guidance message display area 66B switches according to the operating mode of the user device 12. For example, the user device 12 has two operating modes: a viewpoint setting mode for setting the viewpoint and a fixation point setting mode for setting the fixation point. The content displayed in the guidance message display area 66B differs between the viewpoint setting mode and the fixation point setting mode.

[0067] The confirm key 66C and the cancel key 66D are both soft keys. The confirm key 66C is turned on by the user 14 when confirming an instruction received by the reception screen 66. The cancel key 66D is turned on by the user 14 when canceling an instruction received by the reception screen 66.

[0068] The reception screen generation unit 28A acquires multiple captured images 64 from multiple imaging devices 36 and generates an overhead view image 72 based on the acquired multiple captured images 64. The reception screen generation unit 28A then generates reception screen data 70, which shows a reception screen 66 including the overhead view image 72.

[0069] The output unit 28E acquires the reception screen data 70 generated by the reception screen generation unit 28A from the reception screen generation unit 28A and outputs the acquired reception screen data 70 to the transceiver 24. The transceiver 24 transmits the reception screen data 70 input from the output unit 28E to the user device 12. The user device 12 receives the reception screen data 70 transmitted from the transceiver 24 with the transceiver 44 (see Figure 2). The reception screen 66 shown by the reception screen data 70 received by the transceiver 44 (see Figure 2) is displayed on the display 18.

[0070] As an example, as shown in Figure 5, when the operating mode of the user device 12 is the viewpoint setting mode, message 66B1 is displayed in the guidance message display area 66B of the reception screen 66. Message 66B1 is a message prompting the user 14 to specify the viewpoint used to generate the virtual viewpoint image 78 (see Figure 8). Here, the viewpoint refers to a virtual viewpoint for observing the inside of the soccer stadium. The virtual viewpoint refers to the location where a virtual camera that images the subject is installed, rather than the location where an actual camera is installed, such as a physical camera (e.g., imaging device 36) that images the subject.

[0071] The touch panel 20 receives instructions from the user 14 while message 66B1 is displayed in the guidance message display area 66B. In this case, the instructions from the user 14 refer to the instruction of the viewpoint. The viewpoint corresponds to the position of a pixel in the overhead view image 72. The viewpoint is indicated by the user 14 specifying the position of a pixel in the overhead view image 72 via the touch panel 20. The viewpoint may also be a 3D coordinate corresponding to a 3D position in the overhead view image 72. Any method can be used to specify the 3D position. For example, the user 14 may directly input the 3D coordinate position, or the 3D coordinate position may be specified by displaying two images showing the view of the soccer stadium from two planes perpendicular to each other and specifying the pixel position in each image.

[0072] In the example shown in Figure 5, a viewpoint path P1 is shown as an example of a viewpoint, which is a path for observing a subject. The viewpoint path P1 is a collection of multiple viewpoints arranged linearly from a starting point P1s to an ending point P1e. ​​The viewpoint path P1 is defined along the path (in the example shown in Figure 5, a meandering path from the starting point P1s to the ending point P1e) that the user 14 slides (i.e., swipes) with their fingertip 14A on the area of ​​the touch panel 20 corresponding to the display area of ​​the overhead image 72. The observation time from the viewpoint path P1 (for example, the time spent observing between two different viewpoints, and / or the time spent observing while stationary at a single point) is defined by the speed of the slide performed on the touch panel 20 when forming the viewpoint path P1 via the touch panel 20, and the time spent at one viewpoint on the viewpoint path P1 (for example, the time spent with a long press).

[0073] In the example shown in Figure 5, the confirm key 66C is turned on when confirming the viewpoint path P1, and the cancel key 66D is turned on when canceling the viewpoint path P1.

[0074] In the example shown in Figure 5, only viewpoint path P1 is set, but this is merely one example, and multiple viewpoint paths may be set. Furthermore, it is not limited to viewpoint paths; it may also be multiple discontinuous viewpoints, or even just one viewpoint.

[0075] As an example, as shown in Figure 6, when the operating mode of the user device 12 is the fixation point setting mode, message 66B2 is displayed in the guidance message display area 66B of the reception screen 66. Message 66B2 is a message prompting the user 14 to specify the fixation point used to generate the virtual viewpoint image 78 (see Figure 8). Here, the fixation point refers to the point that is virtually fixed on when observing the inside of a soccer stadium from the viewpoint. Once the viewpoint and fixation point are set, the virtual line of sight direction (i.e., the virtual camera imaging direction) is also uniquely determined. The virtual line of sight direction refers to the direction from the viewpoint toward the fixation point.

[0076] The touch panel 20 receives instructions from the user 14 while message 66B2 is displayed in the guidance message display area 66B. In this case, the instructions from the user 14 refer to the instruction of the point of focus. The point of focus corresponds to the position of a pixel in the overhead image 72. The point of focus is indicated by the user 14 indicating the position of a pixel in the overhead image 72 via the touch panel 20. In the example shown in Figure 6, the point of focus GP is shown. The point of focus GP is defined according to the location where the user 14 touches with their fingertip 14A on the area of ​​the touch panel 20 that corresponds to the display area of ​​the overhead image 72. In the example shown in Figure 6, when confirming the point of focus GP, the confirm key 66C is turned on, and when canceling the point of focus GP, the cancel key 66D is turned on. Note that the point of focus may also be a 3D coordinate corresponding to a 3D position in the overhead image 72. Any method can be used to indicate the 3D position, similar to the method for indicating the viewpoint position.

[0077] In the example shown in Figure 6, only the gaze point GP is specified, but this is merely one example. There may be multiple gaze points, or a path in which multiple gaze points are arranged linearly (i.e., a gaze point path). There may be one gaze point path or multiple gaze point paths.

[0078] As an example, as shown in Figure 7, the CPU 52 of the user device 12 generates viewpoint information 74 based on the viewpoint path P1 and the point of focus GP. The viewpoint information 74 is information used to generate a virtual viewpoint image 78 (see Figure 8). The viewpoint information 74 includes total time information 74A, setting completion information 74B, viewpoint path information 74C, required time information 74D, elapsed time information 74E, movement speed information 74F, field of view information 74G, and point of focus information 74H.

[0079] The total time information 74A is information indicating the total time (hereinafter also simply referred to as "total time") during which the virtual viewpoint image 78 (see Figure 8), generated based on one or more viewpoint paths (for example, multiple viewpoint paths including viewpoint path P1) determined in viewpoint setting mode, is played back at the standard playback speed. The total time corresponds to the time spent sliding the fingertip 14A on the touch panel 20 to create multiple viewpoint paths. The total time information 74A is an example of a "time parameter related to time" in the technology of this disclosure.

[0080] The configuration completion information 74B indicates that the configuration of the viewpoint information 74 has been completed. Completion of the configuration of viewpoint information 74 means, for example, that the CPU 52 has completed the generation of viewpoint information 74.

[0081] The viewpoint path information 74C is information indicating the viewpoint path P1 determined in viewpoint setting mode (for example, coordinates that identify the pixel position of the viewpoint path P1 within the overhead view image 72).

[0082] The time required information 74D indicates the time required (hereinafter also simply referred to as "time required") for a first viewpoint observing a subject on the viewpoint path P1 to move from a first position to a second position different from the first position. Here, the first position refers to the starting point P1s (see Figures 5 and 6), and the second position refers to, for example, the ending point P1e (see Figures 5 and 6). Note that this is merely an example, and the first position may be the starting point P1s (see Figures 5 and 6) and the second position may be a viewpoint position in the middle of the viewpoint path P1, or the first position may be a viewpoint position in the middle of the viewpoint path P1 and the second position may be the ending point P1e (see Figures 5 and 6).

[0083] The elapsed time information 74E indicates the position of the second viewpoint observing the subject on the viewpoint path P1, and the elapsed time corresponding to the position of the second viewpoint. The elapsed time corresponding to the position of the second viewpoint (hereinafter also simply referred to as "elapsed time") refers, for example, to the time during which the viewpoint remains stationary at a certain viewpoint position on the viewpoint path P1.

[0084] The movement speed information 74F is information that can identify the movement speed of the position of a third viewpoint observing the subject on the viewpoint path P1. The movement speed of the position of the third viewpoint (hereinafter also simply referred to as "movement speed") refers, for example, to the speed of a slide performed on the touch panel 20 when forming the viewpoint path P1 via the touch panel 20. The movement speed information 74F is associated with each viewpoint within the viewpoint path P1.

[0085] The field of view information 74G is information regarding the field of view (hereinafter also simply referred to as "field of view") for observing the subject on the viewpoint path P1. In this embodiment, the field of view is determined according to the speed of movement. For example, within a range in which an upper limit (e.g., 150 degrees) and a lower limit (e.g., 15 degrees) of the field of view are defined, the field of view becomes narrower as the speed of movement decreases.

[0086] This is merely one example; for instance, the field of view could be narrowed as the speed of movement increases. Alternatively, the field of view could be determined according to the elapsed time. In this case, for example, the field of view could be minimized when the elapsed time exceeds a first predetermined time (e.g., 3 seconds), or maximized when the elapsed time exceeds the first predetermined time.

[0087] Furthermore, the field of view may be determined, for example, according to instructions received by the reception device 50. In this case, the reception device 50 should be able to receive instructions regarding the position of the viewpoint on the viewpoint path P1 where the field of view is to be changed, and the changed field of view.

[0088] The gaze point information 74H is information that can identify the position of the gaze point GP determined in the gaze point setting mode (for example, coordinates that identify the pixel position of the gaze point GP in the overhead view image 72). Note that gaze point information 74H is an example of "gaze point information" related to the technology disclosed herein.

[0089] The CPU 52 outputs viewpoint information 74 to the transceiver 44. The transceiver 44 transmits the viewpoint information 74 received from the CPU 52 to the image processing device 10. The transceiver 24 of the image processing device 10 receives the viewpoint information 74. The viewpoint information acquisition unit 28B of the image processing device 10 acquires the viewpoint information 74 received by the transceiver 24.

[0090] As an example, as shown in Figure 8, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76, which is an image showing the appearance of the subject when observed from a viewpoint specified by the viewpoint information 74, based on a plurality of captured images 64 and viewpoint information 74. For example, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from a plurality of imaging devices 36 according to the viewpoint information 74, and generates a virtual viewpoint image 76 for each viewpoint on the viewpoint path P1 based on the acquired plurality of captured images 64. That is, the virtual viewpoint image generation unit 28C generates a plurality of virtual viewpoint images 76 according to the viewpoint path P1. The plurality of virtual viewpoint images 76 generated according to the viewpoint path P1 refer to the virtual viewpoint images 76 generated for each viewpoint on the viewpoint path P1. The virtual viewpoint image generation unit 28C generates a virtual viewpoint video 78 by arranging the plurality of virtual viewpoint images 76 in chronological order. Furthermore, even if there are viewpoint paths other than viewpoint path P1, the virtual viewpoint image generation unit 28C generates multiple frames of virtual viewpoint images 76 in the same way as viewpoint path P1, and generates a virtual viewpoint video 78 by arranging the generated multiple frames of virtual viewpoint images 76 in chronological order.

[0091] The virtual viewpoint video 78 is a moving image in which multiple frames of virtual viewpoint images 76 are arranged in chronological order. The person who views the virtual viewpoint video 78 is, for example, user 14. The virtual viewpoint video 78 is viewed by user 14 via the display 18 of user device 12. For example, the multiple frames of virtual viewpoint images 76 are displayed on the virtual viewpoint video screen 68 (see Figure 4) of the display 18 of user device 12 at a predetermined frame rate (for example, several tens of frames / second) from the first frame to the last frame, and are viewed by user 14 as the virtual viewpoint video 78.

[0092] As an example, as shown in Figure 9, the screen data generation unit 28D generates ad-included screen data 80 based on viewpoint information 74 acquired by the viewpoint information acquisition unit 28B (for example, viewpoint information 74 used to generate the virtual viewpoint video 78), the virtual viewpoint video 78 generated by the virtual viewpoint image generation unit 28C, and at least one ad video 62 stored in the NVM 30. The ad-included screen data 80 is data for displaying the ad video 62 and the virtual viewpoint video 78 on the display 18 (see Figure 10) of the user device 12 based on the viewpoint information 74. The ad-included screen data 80 also displays the ad video 62 on the display 18 (see Figure 10) at a timing determined according to the viewpoint information 74. to It is also data for display. Note that the ad-containing screen data 80 does not need to be a single data set containing the ad video 62 and the virtual viewpoint video 78; it can be any data that enables the display of the ad video 62 and the virtual viewpoint video 78 on the display 18 of the user device 12 at the timings shown below. For example, the ad-containing screen data 80 may only be data that determines the display timing of the ad video 62 and the virtual viewpoint video 78. In this case, the ad video 62 and the virtual viewpoint video 78 are streamed at the determined timing and displayed on the display 18.

[0093] The ad-containing screen data 80 includes the first screen data 80A, the second screen data 80B, the third screen data 80C, the fourth screen data 80D, the fifth screen data 80E, the sixth screen data 80F, the seventh screen data 80G, and the eighth screen data 80H.

[0094] Here, an example is given in which the ad-containing screen data 80 includes the first screen data 80A, the second screen data 80B, the third screen data 80C, the fourth screen data 80D, the fifth screen data 80E, the sixth screen data 80F, the seventh screen data 80G, and the eighth screen data 80H. However, this is merely an example, and the ad-containing screen data 80 only needs to include one or more of the screen data from the first screen data 80A, the second screen data 80B, the third screen data 80C, the fourth screen data 80D, the fifth screen data 80E, the sixth screen data 80F, the seventh screen data 80G, and the eighth screen data 80H.

[0095] The advertisement-containing screen data 80 is an example of "data" relating to the technology disclosed herein. The first screen data 80A is an example of "first data" relating to the technology disclosed herein. The second screen data 80B is an example of "second data" relating to the technology disclosed herein. The third screen data 80C is an example of "third data" relating to the technology disclosed herein. The fourth screen data 80D is an example of "fourth data" relating to the technology disclosed herein. The fifth screen data 80E is an example of "fifth data" relating to the technology disclosed herein. The sixth screen data 80F is an example of the technology disclosed herein. of This is an example of "Data No. 6" related to the technology. Also, Screen Data No. 7 80G is an example of "Data No. 7" related to the technology of this disclosure. Furthermore, Screen Data No. 8 80H is an example of "Data No. 8" related to the technology of this disclosure.

[0096] The output unit 28E outputs the screen data 80 containing advertisements, generated by the screen data generation unit 28D, to the transmitting / receiving device 24. The transmitting / receiving device 24 transmits the screen data 80 containing advertisements, received from the output unit 28E, to the user device 12.

[0097] As an example, as shown in Figure 10, various types of information are exchanged between the image processing device 10 and the user device 12. For example, the user device 12 transmits viewpoint information 74 to the image processing device 10. The image processing device 10 transmits reception screen data 70 to the user device 12. The image processing device 10 also receives the viewpoint information 74 transmitted from the user device 12 and generates advertisement-containing screen data 80 based on the received viewpoint information 74. Then, the image processing device 10 transmits the advertisement-containing screen data 80 to the user device 12.

[0098] The user device 12 receives reception screen data 70 and advertisement-containing screen data 80 transmitted from the image processing device 10. The user device 12 displays the reception screen 66 indicated by the reception screen data 70 on the display 18, and displays the image based on the advertisement-containing screen data 80 on the virtual viewpoint video screen 68. Here, the image based on the advertisement-containing screen data 80 refers to the advertisement-containing virtual viewpoint video 81. The advertisement-containing virtual viewpoint video 81 refers to a video that includes the advertisement video 62 and the virtual viewpoint video 78. In this embodiment, the advertisement-containing virtual viewpoint video 81 is classified into advertisement-containing virtual viewpoint videos 81A to 81H (see Figures 11, 12, and 14 to 19).

[0099] As an example, as shown in Figure 11, the screen data generation unit 28D generates first screen data 80A, which includes a virtual viewpoint video with an advertisement 81A, based on total time information 74A, advertising video 62, and virtual viewpoint video 78. The first screen data 80A is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The first screen data 80A is also data that causes the advertising video 62 to be displayed on the display 18 in accordance with the total time information 74A.

[0100] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81A by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at a timing determined according to the total time information 74A.

[0101] A first example of timing determined according to the total time information 74A is the time interval obtained by dividing the total time indicated by the total time information 74A into equal parts. A second example of timing determined according to the total time information 74A is the timing before the first frame of the virtual viewpoint image 76 included in the virtual viewpoint video 78 is displayed, if the total time indicated by the total time information 74A is less than the second predetermined time (for example, 20 seconds). A third example of timing determined according to the total time information 74A is the timing during the display of the virtual viewpoint video 78, if the total time indicated by the total time information 74A is equal to or greater than the second predetermined time. Another example of timing determined according to the total time information 74A is the timing after the final frame of the virtual viewpoint image 76 is displayed.

[0102] As an example, as shown in Figure 12, the screen data generation unit 28D generates second screen data 80B, which includes a virtual viewpoint video with an advertisement 81B, based on the setting completion information 74B, the advertisement video 62, and the virtual viewpoint video 78. The second screen data 80B is data that causes the virtual viewpoint video 78 to be displayed on the display 18. In addition, the second screen data 80B is also data that causes the advertisement video 62 to be displayed on the display 18 between the completion of the setting of viewpoint information 74 and the display of the virtual viewpoint video 78 on the display 18, according to the setting completion information 74B.

[0103] The screen data generation unit 28D determines whether or not the viewpoint information 74 contains the setting completion information 74B. If the screen data generation unit 28D determines that the viewpoint information 74 contains the setting completion information 74B, it generates an advertisement-included virtual viewpoint video 81B by inserting the advertisement video 62 into the virtual viewpoint video 78 after the setting of the viewpoint information 74 is complete and before the first frame of the virtual viewpoint image 76 included in the virtual viewpoint video 78 is displayed on the display 18.

[0104] In the example shown in Figure 12, the advertisement video 62 is also inserted between the first frame's virtual viewpoint image 76 and the last frame's virtual viewpoint image 76 of the virtual viewpoint video 78. The method for inserting the advertisement video 62 between the first frame's virtual viewpoint image 76 and the last frame's virtual viewpoint image 76 may be the same as the method used to generate the advertisement-filled virtual viewpoint video 81A shown in Figure 11, or it may be inserted in the middle of the virtual viewpoint video 78 by another method (for example, the same method used to generate the advertisement-filled virtual viewpoint videos 81C to 81H described later).

[0105] As an example, as shown in Figure 13, if multiple viewpoint paths, including viewpoint path P1, are specified via the reception screen 66, and virtual viewpoint images generated based on multiple starting paths are played back consecutively, then from the end point of one viewpoint path to the next... Viewpoint path The continuity of viewpoint information 74 may be interrupted between the start point and the beginning point. In the example shown in Figure 13, in addition to viewpoint path P1, viewpoint path P2 from the start point P2s to the end point P2e is specified by the user 14, and a virtual viewpoint image based on viewpoint path P1 is played, followed by a virtual viewpoint image based on viewpoint path P2. Here, the end point P1e of viewpoint path P1 and the start point P2s of viewpoint path P2 are discontinuous. That is, the continuity of viewpoint information 74 is interrupted between the end point P1e of viewpoint path P1 and the start point P2s of viewpoint path P2. Therefore, when the viewpoint moves from the end point P1e to the start point P2s, the virtual viewpoint image changes significantly. In this case, the screen data generation unit 28D generates third screen data 80C.

[0106] As an example, as shown in Figure 14, the screen data generation unit 28D generates a third screen data 80C, which includes an advertisement-filled virtual viewpoint video 81C, based on the advertisement video 62, viewpoint path information 74C, and virtual viewpoint video 78. The third screen data 80C is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The third screen data 80C is also data that causes the advertisement video 62 to be displayed on the display 18 in accordance with the timing when the continuity of the viewpoint information 74 is interrupted.

[0107] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81C by referring to the viewpoint path information 74C and inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at the moment when the continuity of the viewpoint is interrupted. Here, the moment when the continuity of the viewpoint is interrupted refers, for example, to the time from when the viewpoint corresponding to the virtual viewpoint image 76 reaches the endpoint P1e of viewpoint path P1 (see Figure 14) until before it reaches the starting point P2s of viewpoint path P2 (see Figure 14). In other words, in this example, the screen data generation unit 28D determines that the continuity of the viewpoint is interrupted because the viewpoint path is divided into two by the instruction of the user 14. Note that the moment when the continuity of the viewpoint is interrupted is not limited to this moment. For example, the screen data generation unit 28D may determine that the continuity of the viewpoint is interrupted when the distance between two temporally continuous viewpoints exceeds a predetermined threshold, or when the content of the virtual viewpoint video, which is determined by the position of the viewpoint and the position of the point of focus, changes significantly.

[0108] As an example, as shown in Figure 15, the screen data generation unit 28D generates a fourth screen data 80D, which includes an advertisement-filled virtual viewpoint video 81D, based on the advertisement video 62, viewpoint path information 74C, and virtual viewpoint video 78. The fourth screen data 80D is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The fourth screen data 80D is also data that causes the advertisement video 62 to be displayed on the display 18 at intervals defined by the viewpoint path P1 indicated by the viewpoint path information 74C.

[0109] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81D by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at position intervals obtained by dividing the viewpoint path P1 indicated by the viewpoint path information 74C into N (a natural number of 2 or more) parts. The number of divisions of the viewpoint path P1 may be determined according to instructions received by the reception device 50 (see Figure 2), or it may be determined according to the length from the starting point P1s to the ending point P1e of the viewpoint path P1.

[0110] Furthermore, the viewpoint path P1 does not necessarily have to be divided into N equal parts. For example, the advertising video 62 may be displayed on the display 18 at positional intervals obtained by dividing the viewpoint path P1 into N parts, such that the insertion interval of the advertising video 62 shortens as the viewpoint corresponding to the virtual viewpoint image 76 approaches the endpoint P1e of the viewpoint path P1, that is, as it approaches the virtual viewpoint image 76 of the final frame of the virtual viewpoint video 78.

[0111] As an example, as shown in Figure 16, the screen data generation unit 28D generates a fifth screen data 80E, which includes an advertisement-filled virtual viewpoint video 81E, based on the advertisement video 62, the required time information 74D, and the virtual viewpoint video 78. The fifth screen data 80E is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The fifth screen data 80E is also data that causes the advertisement video 62 to be displayed on the display 18 at intervals defined by the required time information 74D.

[0112] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81E by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at regular time intervals in which the required time indicated by the required time information 74D is divided equally. The number of equal divisions of the required time may be determined according to the instructions received by the reception device 50 (see Figure 2), or according to the length of the required time indicated by the required time information 74D. Furthermore, equal division is merely an example; for example, the insertion interval of the advertisement video 62 may be shortened as the viewpoint corresponding to the virtual viewpoint image 76 approaches the endpoint P1e of the viewpoint path P1, that is, as it approaches the virtual viewpoint image 76 of the final frame of the virtual viewpoint video 78.

[0113] As an example, as shown in Figure 17, the screen data generation unit 28D generates a sixth screen data 80F, which includes an advertisement-filled virtual viewpoint video 81F, based on the advertisement video 62, elapsed time information 74E, and virtual viewpoint video 78. The sixth screen data 80F is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The sixth screen data 80F is also data that causes the advertisement video 62 to be displayed on the display 18 at a timing determined according to the relationship between the elapsed time indicated by the elapsed time information 74E and the position of the second viewpoint.

[0114] The screen data generation unit 28D refers to the elapsed time information 74E and generates an advertisement-included virtual viewpoint video 81F so that the advertisement video 62 is displayed on the display 18 at a position where the viewpoint is stationary. For example, the screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81F by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at a viewpoint position among the multiple viewpoints included in the viewpoint path P1 where the elapsed time indicated by the elapsed time information 74E is equal to or greater than a first threshold (e.g., 5 seconds) (e.g., at the timing when the elapsed time is equal to or greater than the first threshold). The first threshold may be a fixed value or a variable value that is changed according to instructions received by the reception device 50 and / or various conditions.

[0115] As an example, as shown in Figure 18, the screen data generation unit 28D generates a seventh screen data 80G, which includes an advertisement-filled virtual viewpoint video 81G, based on the advertisement video 62, movement speed information 74F, and virtual viewpoint video 78. The seventh screen data 80G is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The seventh screen data 80G is also data that causes the advertisement video 62 to be displayed on the display 18 when the movement speed, determined from the movement speed information 74F, falls below a second threshold. The second threshold is an example of a "threshold" related to the technology of this disclosure.

[0116] The screen data generation unit 28D determines the movement speed identified from the movement speed information 74F among the multiple viewpoints included in the viewpoint path P1 as a second threshold (e.g., 5 mm / second). That's all. The virtual viewpoint video 81F containing an advertisement is generated by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at a viewpoint position that satisfies the conditions (for example, when the movement speed satisfies the condition of being equal to or greater than the second threshold). The second threshold may be a fixed value, or it may be a variable value that is changed according to instructions received by the receiving device 50 and / or various conditions.

[0117] As an example, as shown in Figure 19, the screen data generation unit 28D generates eighth screen data 80H, which includes an advertisement-filled virtual viewpoint video 81H, based on the advertisement video 62, field of view information 74G, and virtual viewpoint video 78. The eighth screen data 80H is data that causes the virtual viewpoint video 78 to be displayed on the display 18. The eighth screen data 80H is also data that causes the advertisement video 62 to be displayed on the display 18 at a timing determined according to the field of view information 74G.

[0118] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81H by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 at the position of a viewpoint where the field of view indicated by the field of view information 74G among the multiple viewpoints included in the viewpoint path P1 falls below a third threshold (for example, 40 degrees) (for example, at the timing when the field of view indicated by the field of view information 74G falls below the third threshold).

[0119] Here, an example is given in which an advertisement-included virtual viewpoint video 81H is generated such that the advertisement video 62 is displayed on the display 18 when the field of view indicated by the field of view information 74G falls below a third threshold, but the technology of this disclosure is not limited to this. For example, the advertisement-included virtual viewpoint video 81H may be generated such that the advertisement video 62 is displayed on the display 18 when the field of view indicated by the field of view information 74G falls above a fourth threshold (e.g., 110 degrees), or the advertisement-included virtual viewpoint video 81H may be generated such that the advertisement video 62 is displayed on the display 18 when the field of view indicated by the field of view information 74G changes.

[0120] The third and fourth threshold values ​​may be fixed values, or they may be variable values ​​that are changed according to instructions received by the receiving device 50 and / or various conditions.

[0121] Next, the operation of the image processing device 10 will be explained with reference to Figure 20.

[0122] Figure 20 shows an example of the screen generation process performed by the CPU 28 of the image processing device 10. The screen generation process shown in Figure 20 relates to the technology of this disclosure. 「 Image processing method 」 This is one example.

[0123] In the screen generation process shown in Figure 20, first, in step ST10, the reception screen generation unit 28A generates reception screen data 70 based on multiple captured images 64 (see Figure 4). After the process in step ST10 is executed, the screen generation process moves on to step ST12.

[0124] In step ST12, the output unit 28E transmits the reception screen data 70 generated by the reception screen generation unit 28A to the user device 12 via the transmitting / receiving device 24. After the processing in step ST12 is completed, the screen generation process proceeds to step ST14.

[0125] When the processing in step ST12 is executed, the reception screen data 70 is transmitted from the image processing device 10 to the user device 12. The user device 12 receives the reception screen data 70 and displays the reception screen 66 indicated by the received reception screen data 70 on the display 18 (see Figures 4 to 6). When the reception screen 66 is displayed on the display 18 of the user device 12, the user 14 provides instructions such as the viewpoint and gaze point to the user device 12 via the touch panel 20 (see Figures 5 and 6). The CPU 52 of the user device 12 generates viewpoint information 74 based on the viewpoint and gaze point received by the touch panel 20 and transmits the generated viewpoint information 74 to the image processing device 10 via the transmitting / receiving device 44 (see Figure 7).

[0126] In step ST14, the viewpoint information acquisition unit 28B determines whether or not viewpoint information 74 has been received by the transmitting / receiving device 24. If viewpoint information 74 has not been received by the transmitting / receiving device 24 in step ST14, the determination is denied, and the screen generation process proceeds to step ST22. If viewpoint information 74 has been received by the transmitting / receiving device 24 in step ST14, the determination is affirmed, and the screen generation process proceeds to step ST16. The viewpoint information acquisition unit 28B acquires viewpoint information 74 received by the transmitting / receiving device 24 (see Figure 7).

[0127] In step ST16, the virtual viewpoint image generation unit 28C acquires multiple captured images 64 from multiple imaging devices 36 according to the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B. Then, the virtual viewpoint image generation unit 28C generates a virtual viewpoint video 78 based on the multiple captured images 64 acquired from the multiple imaging devices 36 and the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B. After the processing in step ST16 is completed, the screen generation process moves on to step ST18.

[0128] In step ST18, the screen data generation unit 28D generates screen data 80 containing advertisements based on the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B (the viewpoint information 74 used to generate the virtual viewpoint image 78 in step ST16), the virtual viewpoint image 78 generated in step ST16, and the advertisement image 62 stored in the NVM 30 (see Figures 9 to 19). After the processing in step ST18 is completed, the screen generation process moves on to step ST20.

[0129] In step ST20, the output unit 28E transmits the advertisement-containing screen data 80 generated in step ST18 to the user device 12 via the transmitting / receiving device 24 (see Figure 9).

[0130] When the process in step ST20 is executed, the image processing device 10 transmits the advertisement-containing screen data 80 to the user device 12. The user device 12 receives the advertisement-containing screen data 80 and displays the advertisement-containing virtual viewpoint video 81 contained in the received advertisement-containing screen data 80 on the virtual viewpoint video screen 68 of the display 18 (see Figure 10). The advertisement-containing virtual viewpoint video 81 displayed on the virtual viewpoint video screen 68 is viewed by the user 14.

[0131] The virtual viewpoint video 81 with advertisements displayed on the virtual viewpoint video screen 68 may be one or more of the virtual viewpoint videos 81A to 81H with advertisements described above. Whether any of the virtual viewpoint videos 81A to 81H with advertisements are displayed on the virtual viewpoint video screen 68 of the display 18 may be determined randomly, according to the content of the viewpoint information 74, or according to the number of times the user 14 has viewed the virtual viewpoint video 78.

[0132] In step ST22, the output unit 28E determines whether the conditions for terminating the screen generation process (hereinafter referred to as "termination conditions") have been met. A first example of a termination condition is that an instruction to terminate the screen generation process has been received by the receiving device 50 (see Figure 2). A second example of a termination condition is that communication between the image processing device 10 and one or more predetermined imaging devices 36 among the multiple imaging devices 36 has been interrupted. A third example of a termination condition is that a predetermined time (for example, 60 seconds) has elapsed without the determination being affirmed in step ST14.

[0133] In step ST22, if the termination condition is not met, the judgment is denied, and the screen generation process proceeds to step ST10. In step ST22, if the termination condition is met, the judgment is affirmed, and the screen generation process ends.

[0134] As described above, in the image processing device 10, the advertising video 62, which was created in a different process from the virtual viewpoint video 78, and the virtual viewpoint video 78 are displayed on the display 18 of the user device 12 based on viewpoint information 74. Specifically, the virtual viewpoint video 78 is displayed on the display 18, and the advertising video 62 is displayed on the display 18 at a timing determined according to the viewpoint information 74. Therefore, with this configuration, the advertising video 62 can be shown to the user 14, who is the viewer of the virtual viewpoint video 78.

[0135] Furthermore, the image processing device 10 generates first screen data 80A. The first screen data 80A is data for displaying the virtual viewpoint video 78 on the display 18. The first screen data 80A is also data for displaying the advertising video 62 on the display 18 according to the total time information 74A. Therefore, with this configuration, compared to the case where the advertising video 62 is displayed on the display 18 based solely on the viewpoint, it is easier for the provider of the advertising video 62 to display the advertising video 62 on the display 18 at a timing convenient for the user 14, and / or for the user 14.

[0136] Furthermore, the image processing device 10 generates second screen data 80B. The second screen data 80B is data for displaying the virtual viewpoint video 78 on the display 18. In addition, according to the setting completion information 74B, the second screen data 80B is also data for displaying the advertisement video 62 on the display 18 between the completion of the viewpoint information 74 settings and the display of the virtual viewpoint video 78 on the display 18. Therefore, with this configuration, the advertisement video 62 can be shown to the user 14 between the completion of the viewpoint information 74 settings and the user 14 viewing the virtual viewpoint video 78.

[0137] Furthermore, the image processing device 10 generates third screen data 80C. The third screen data 80C is data for displaying the virtual viewpoint video 78 on the display 18. The third screen data 80C is also data for displaying the advertising video 62 on the display 18 in accordance with the timing when the continuity of the viewpoint information 74 is interrupted. Therefore, according to this configuration 、 The advertisement video 62 can be shown to user 14 at the moment when the continuity of viewpoint information 74 is interrupted.

[0138] Furthermore, the image processing device 10 generates fourth screen data 80D. The fourth screen data 80D is data for displaying the virtual viewpoint video 78 on the display 18. The fourth screen data 80D is also data for displaying the advertising video 62 on the display 18 at intervals defined by the viewpoint path P1 indicated by the viewpoint path information 74C. Therefore, with this configuration, the user 14 can be shown the advertising video 62 at intervals defined by the viewpoint path P1.

[0139] Furthermore, the image processing device 10 generates fifth screen data 80E. Fifth screen data 80E is data that displays the virtual viewpoint video 78 on the display 18. Also, the fifth screen data 80E displays the advertising video 62 at intervals where the required time is divided according to the required time information 74D. ofThis is also data to be displayed on the display 18. Therefore, with this configuration, the user 14 can be shown the advertising video 62 at intervals that are divided by the time required for the viewpoint to move from the first position to the second position.

[0140] Furthermore, the image processing device 10 generates sixth screen data 80F. The sixth screen data 80F is data for displaying the virtual viewpoint video 78 on the display 18. The sixth screen data 80F is also data for displaying the advertisement video 62 on the display 18 at a timing determined according to the relationship between the elapsed time indicated by the elapsed time information 74E and the position of the second viewpoint. Therefore, with this configuration, the advertisement video 62 can be shown to the user 14 at a timing determined according to the relationship between the elapsed time indicated by the elapsed time information 74E and the position of the second viewpoint.

[0141] Furthermore, the image processing device 10 generates seventh screen data 80G. The seventh screen data 80G is data for displaying the virtual viewpoint video 78 on the display 18. The seventh screen data 80G is also data for displaying the advertisement video 62 on the display 18 when the movement speed determined from the movement speed information 74F falls below the second threshold. Therefore, with this configuration, the advertisement video 62 can be shown to the user 14 when the movement speed determined from the movement speed information 74F falls below the second threshold.

[0142] Furthermore, the image processing device 10 generates eighth screen data 80H. The eighth screen data 80H is data for displaying the virtual viewpoint video 78 on the display 18. The eighth screen data 80H is also data for displaying the advertising video 62 on the display 18 at a timing determined according to the field of view information 74G. Therefore, with this configuration, the advertising video 62 can be shown to the user 14 at a timing determined according to the field of view information 74G.

[0143] In the above embodiment, the advertising video 62 is exemplified as an image created by a process different from that of the virtual viewpoint image 78 (see Figure 8, etc.), but the technology of this disclosure is not limited thereto. For example, the advertising video 62 may be an image created without using multiple captured images 64, and in this case as well, the same effect as in the above embodiment can be obtained. Also, the advertising video 62 may be an image created by a process different from that of the virtual viewpoint image 76 (see Figure 8, etc.) and without using multiple captured images 64 (see Figures 3, 4 and 8, etc.). Furthermore, the advertising video 62 may be an image created using at least a portion of the multiple captured images 64. Also, the advertising video 62 may be an image that is not affected by the multiple imaging devices 36, an image that does not depend on the RAW image obtained by the subject being imaged by the multiple imaging devices 36, or an image that is unrelated to the data obtained from the multiple imaging devices 36.

[0144] Furthermore, although the above embodiment described an example in which an advertising video 62 is inserted between virtual viewpoint images 76, the technology of this disclosure is not limited thereto. For example, the advertising video 62 may be superimposed on the virtual viewpoint image 78, or the advertising image may be embedded in the virtual viewpoint image 76 and displayed. In this case, as shown in Figure 21 as an example, the screen data generation unit 28D generates a ninth screen data 80I that includes an advertising-filled virtual viewpoint image 81I. The ninth screen data 80I is an example of the "ninth data" relating to the technology of this disclosure.

[0145] The ninth screen data 80I is data for displaying the virtual viewpoint video 78 on the display 18. Furthermore, the ninth screen data 80I is also data for displaying the advertisement video 62 on the display 18 at the same time that the display 18 begins displaying the virtual viewpoint video 78.

[0146] The screen data generation unit 28D generates an advertisement-included virtual viewpoint video 81I by superimposing the advertisement video 62 onto the virtual viewpoint image 78 so that the advertisement video 62 is superimposed onto a predetermined number of virtual viewpoint images 76 (for example, several hundred frames) starting from the first frame of the virtual viewpoint image 76, so that the advertisement video 62 is displayed on the display 18 when the display 18 starts displaying the virtual viewpoint video 78. The superimposition is achieved, for example, by alpha blending. The advertisement video 62 may be superimposed onto the entirety of each virtual viewpoint image 76 included in the advertisement-included virtual viewpoint video 81I, or it may be superimposed onto a part of each virtual viewpoint image 76.

[0147] Furthermore, the screen data generation unit 28D may generate an advertisement-included virtual viewpoint video 81I by embedding multiple advertisement images included in the advertisement video 62 sequentially from the first frame into a predetermined number of virtual viewpoint images 76 (for example, several hundred frames) starting from the first frame, so that the advertisement video 62 is displayed on the display 18 when the display 18 starts displaying the virtual viewpoint video 78. The advertisement images can be embedded in a part of the virtual viewpoint image 76.

[0148] The virtual viewpoint video 81I with advertisements generated in this way is displayed on the display 18 of the user device 12. Therefore, with this configuration, the advertisement video 62 can be shown to the user 14, who is viewing the virtual viewpoint video 78, at the time the display of the virtual viewpoint video 78 begins. Alternatively, the virtual viewpoint video 81I with advertisements may be generated by sequentially embedding multiple advertisement images included in the advertisement video 62, starting from virtual viewpoint images 76 that are a predetermined number of frames earlier in time from the final frame virtual viewpoint image 76, up to the final frame virtual viewpoint image 76. With this configuration, the advertisement video 62 can be shown to the user 14, who is viewing the virtual viewpoint video 78, around the time the display of the virtual viewpoint video 78 ends.

[0149] Furthermore, in the example shown in Figure 21, a ninth screen data 80I is shown for displaying an advertisement video 62 on the display 18 at the timing when the display of the virtual viewpoint video 78 on the display 18 begins, but the technology of this disclosure is not limited to this. For example, as shown in Figure 22, screen data 80J including an advertisement-containing virtual viewpoint video 81J may be used instead of the ninth screen data 80I. Screen data 80J is an example of the "ninth data" relating to the technology of this disclosure.

[0150] The screen data 80J is data for displaying the virtual viewpoint video 78 on the display 18. The screen data 80J is also data for displaying the virtual viewpoint video 78 on the display 18 when the display of the advertisement video 62 on the display 18 has finished. The screen data generation unit 28D generates the advertisement-included virtual viewpoint video 81J by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 before the display of the virtual viewpoint video 78 begins, and the virtual viewpoint video 78 is displayed on the display 18 when the display of the advertisement video 62 has finished.

[0151] The virtual viewpoint video 81J containing advertisements, generated in this manner, is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the advertisement video 62 can be shown to the user 14, who is the viewer of the virtual viewpoint video 78, before the display of the virtual viewpoint video 78 begins. Similarly, the advertisement video 62 may be shown to the user 14, who is the viewer of the virtual viewpoint video 78, after the display of the virtual viewpoint video 78 has ended.

[0152] Furthermore, at least one of the ninth screen data 80I and screen data 80J may be included in the advertisement-containing screen data 80 (see Figure 9) described in the above embodiment.

[0153] Furthermore, the output unit 28E (see Figure 4) may transmit at least one of the ninth screen data 80I and screen data 80J to the user device 12 via the transmitting / receiving device 24, provided that the reception of viewpoint information 74 by the reception screen 66 indicated by the reception screen data 70 (see Figures 4 to 7) is completed. Note that the reception screen data 70 is an example of the "tenth data" related to the technology of this disclosure.

[0154] When the ninth screen data 80I is transmitted to the user device 12, the virtual viewpoint video 81I containing advertisements, which is included in the ninth screen data 80I, is displayed on the user device 12's display 18. When screen data 80J is transmitted to the user device 12, the virtual viewpoint video 81J containing advertisements, which is included in the screen data 80J, is displayed on the user device 12's display 18. Therefore, with this configuration, the advertisement video 62 can be shown to the user 14, who is a viewer of the virtual viewpoint video 78, at the moment the display of the virtual viewpoint video 78 begins, provided that the reception of viewpoint information 74 by the reception screen 66 is complete.

[0155] Furthermore, although a smartphone was given as an example of the user device 12 in the above embodiment, the technology of this disclosure is not limited to this, and as an example, as shown in Figure 23, the technology of this disclosure can also be applied by using an HMD 116 instead of the user device 12. In the example shown in Figure 23, the HMD 116 comprises a main body 116A and a mounting part 116B. When the HMD 116 is attached to the user 14, the main body 116A is located in front of the user 14's eyes, and the mounting part 116B is located on the upper half of the user 14's head. The mounting part 116B is a strip-shaped member with a width of several centimeters and is fixed in close contact with the upper half of the user 14's head.

[0156] The main unit 116A is equipped with various electrical devices. These electrical devices include a transceiver corresponding to the transceiver 44 of the user device 12, a display body corresponding to the display 18 of the user device 12, a gyro sensor 118, and a computer 120 corresponding to the computer 40 of the user device 12.

[0157] The HMD116 is used in conjunction with the user device 12. That is, the HMD116 is connected to the user device 12 in a communicative manner, and various types of information are exchanged between the HMD116 and the user device 12.

[0158] The HMD116's display unit shows a reception screen 66 and a virtual viewpoint video screen 68, similar to the display 18 of the user device 12. The reception screen 66 displays, for example, the viewpoint (e.g., viewpoint path P1) and the gaze point GP (see Figure 6) set by the user 14 using the user device 12.

[0159] The gyro sensor 118 detects the angular velocity of the HMD 116. The CPU 52 of the user device 12 modifies the content of the gaze point information 74H included in the viewpoint information 74. That is, the gaze point GP indicated by the gaze point information 74H is changed according to the angular velocity detected by the gyro sensor 118. For example, if the user 14 turns their head to the right at a speed of 1 m / s or more, the CPU 52 of the user device 12 controls the display 18 and HMD 116 to displace the gaze point GP in the reception screen 66 to the right by a predetermined amount (for example, a few millimeters), and rewrites the gaze point information 74H with information indicating the position of the gaze point GP after the displacement.

[0160] The screen data generation unit 28D generates screen data 80K, which includes the advertisement-filled virtual viewpoint video 81K. The screen data generation unit 28D controls the display timing of the advertisement video 62 according to the fluctuations in the gaze point GP (see Figure 6) identified from the gaze point information 74H. For example, the screen data generation unit 28D generates the advertisement-filled virtual viewpoint video 81K by inserting the advertisement video 62 into the virtual viewpoint video 78 so that the advertisement video 62 is displayed on the display 18 of the user device 12 and the display body of the HMD 116 when the gaze point GP changes abruptly, that is, when the user 14 turns their head to the right or left at a speed greater than a predetermined value.

[0161] As a result, the advertising video 62 is displayed on the display 18 of the user device 12 and the display body of the HMD 116 when the user 14 turns their head to the right or left at a speed greater than or equal to a predetermined value. Therefore, with this configuration, the user 14, who is the viewer of the virtual viewpoint video 78, can be shown an advertising video 62 whose display timing changes according to the fluctuations of the gaze point GP (see Figure 6).

[0162] In this explanation, we have used an example where user 14 turns their head to the right or left at a speed greater than the default value, but the direction of the head turn may be other than to the right or left. Also, if the user device 12 is equipped with a gyro sensor and / or an accelerometer, the gaze point GP may be changed in the same manner as with the HMD 116 by shaking the user device 12. In this case as well, the display timing of the advertisement video 62 may be controlled according to the changes in the gaze point GP in the same manner as with the HMD 116.

[0163] Furthermore, while this description has given an example of how the advertising video 62 is displayed on the display 18 of the user device 12 and the display body of the HMD 116 at the moment when the gaze point GP changes abruptly, the technology of this disclosure is not limited to this. For example, the advertising video 62 may be displayed on the display 18 of the user device 12 and the display body of the HMD 116 at the moment when the gaze point GP stops moving after a sudden change.

[0164] Furthermore, although this explanation describes an example where the virtual viewpoint video 81K containing advertisements is displayed on both the display 18 of the user device 12 and the display body of the HMD 116, the technology of this disclosure is not limited to this, and it is sufficient to display the virtual viewpoint video 81K containing advertisements on at least the display body of the HMD 116 among the display 18 of the user device 12 and the display body of the HMD 116.

[0165] Furthermore, although this explanation describes an example of using both the user device 12 and the HMD 116, screen data 80K may be generated by either the user device 12 or the HMD 116, and the generated screen data 80K may be transmitted to the image processing device 10.

[0166] Furthermore, although an example of how a sudden change in the gaze point GP is detected using the gyro sensor 118 has been described here, the technology of this disclosure is not limited to this, and a sudden change in the gaze point GP may be detected using an accelerometer and / or an eye tracker together with the gyro sensor 118, or in place of the gyro sensor 118. When an eye tracker is used, the movement of the user's 14 eyeballs is detected, and the advertising video 62 is displayed on at least the display body of the HMD 116, which is one of the display 18 of the user device 12 and the display body of the HMD 116, at the timing when it is detected that the eyeballs have moved by a specific amount (e.g., 5 millimeters) within a specific time (e.g., 1 second).

[0167] In the above embodiment, an example of a configuration in which the screen generation process is performed by the image processing device 10 was described, but the screen generation process may be performed by multiple devices in a distributed manner. In this case, for example, as shown in Figure 24, an image processing system 200 is used instead of the image processing system 2 described in the above embodiment. The image processing system 200 includes a user device 12, a first image processing device 202, and a second image processing device 204, which are connected to each other in a communicative manner. The first image processing device 202 and the second image processing device 204 are examples of "image processing devices" related to the technology of this disclosure.

[0168] The electrical hardware configuration of the first image processing device 202 is the same as that of the image processing device 10 described in the above embodiment. The CPU 28 of the first image processing device 202 operates as a reception screen generation unit 28A, a viewpoint information acquisition unit 28B, and a virtual viewpoint image generation unit 28C, similar to the above embodiment. The viewpoint information acquisition unit 28B transmits viewpoint information 74 (see Figures 7 and 8) to the second image processing device 204, and the virtual viewpoint image generation unit 28C transmits virtual viewpoint video 78 to the second image processing device 204. The second image processing device 204 receives the viewpoint information 74 transmitted from the viewpoint information acquisition unit 28B and the virtual viewpoint video 78 transmitted from the virtual viewpoint image generation unit 28C.

[0169] The second image processing device 204 includes a computer with a CPU 206, an NVM 208, and RAM 210. The NVM 208 stores multiple advertising videos 62. The CPU 206 operates as the screen data generation unit 28D and output unit 28E described in the above embodiment. The screen data generation unit 28D generates advertising-included screen data 80 (see Figure 9) based on viewpoint information 74, virtual viewpoint video 78, and advertising video 62. The output unit 28E transmits the advertising-included screen data 80 generated by the screen data generation unit 28D to the user device 12. The display 18 of the user device 12 displays the advertising-included virtual viewpoint video 81 (see Figure 10) contained in the advertising-included screen data 80, and the advertising-included virtual viewpoint video 81 is viewed by the user 14. Therefore, even when the screen generation process is distributed between the first image processing device 202 and the second image processing device 204, the effects described in the above embodiment can be obtained.

[0170] Furthermore, in the above embodiment, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76, which is an image showing the appearance of the subject when it is observed from a viewpoint specified by the viewpoint information 74, based on a plurality of captured images 64 and viewpoint information 74. However, the technology of this disclosure is not limited thereto, and the virtual viewpoint image generation unit 28C may cause an external device (e.g., a server) that is communicably connected to the image processing device 10 to generate the virtual viewpoint image 76, and acquire the virtual viewpoint image 76 from the external device.

[0171] Furthermore, although a computer 22 is used as an example in the above embodiment, the technology of this disclosure is not limited thereto. For example, a device including an ASIC, FPGA, and / or PLD may be used instead of the computer 22. Alternatively, a combination of hardware and software configurations may be used instead of the computer 22. The same applies to the computer 40 of the user device 12.

[0172] Furthermore, although the screen generation processing program 38 is stored in the NVM 30 in the above embodiment, the technology of this disclosure is not limited thereto. For example, as shown in Figure 25, the screen generation processing program 38 may be stored in any portable storage medium 300, such as an SSD or USB memory, which is a non-temporary storage medium. In this case, the screen generation processing program 38 stored in the storage medium 300 is installed in the computer 22, and the CPU 28 executes the screen generation processing according to the screen generation processing program 38.

[0173] Alternatively, the screen generation processing program 38 may be stored in the memory of another computer or server device connected to the computer 22 via a communication network (not shown), and the screen generation processing program 38 may be downloaded to the image processing device 10 upon request from the image processing device 10. In this case, the screen generation process is executed by the CPU 28 of the computer 22 according to the downloaded screen generation processing program 38.

[0174] Furthermore, although the above embodiment illustrates a CPU 28, at least one other CPU, at least one GPU, and / or at least one TPU may be used instead of, or in conjunction with, the CPU 28.

[0175] The following types of processors can be used as hardware resources to perform screen generation processing. For example, a CPU is a general-purpose processor that functions as a hardware resource to perform screen generation processing according to software, i.e., a program, as described above. Other processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which have circuit configurations specifically designed to perform specialized processing. All of these processors have built-in or connected memory, and all of them use memory to perform screen generation processing.

[0176] The hardware resources that perform the screen generation process may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform the screen generation process may consist of a single processor.

[0177] Examples of configurations using a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, as exemplified by client and server computers, and this processor functions as a hardware resource that performs screen generation processing. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform screen generation processing, on a single IC chip, as exemplified by SoCs. Thus, screen generation processing is realized using one or more of the above types of processors as hardware resources.

[0178] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices.

[0179] Furthermore, the screen generation process described above is merely an example. Therefore, it goes without saying that you may remove unnecessary steps, add new steps, or change the processing order, as long as you do not deviate from the main purpose.

[0180] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0181] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0182] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data for displaying a specific image created in a process different from the virtual viewpoint image and the virtual viewpoint image on a display based on the viewpoint information, wherein the data for displaying the specific image on the display is output at a timing determined according to the viewpoint information. Image processing device.

2. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data for displaying a specific image created without using the aforementioned plurality of captured images and the virtual viewpoint image on a display based on the viewpoint information, wherein the data for displaying the specific image on the display is output at a timing determined according to the viewpoint information. Image processing device.

3. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. The system outputs data for displaying the virtual viewpoint image on the display, and also outputs data for displaying a specific image on the display at a timing determined according to the viewpoint information. Image processing device.

4. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created in a process different from the aforementioned virtual viewpoint image and the aforementioned virtual viewpoint image on a display based on the viewpoint information. The viewpoint information includes setting completion information indicating that the setting of the viewpoint information has been completed. The aforementioned data is second data for displaying the virtual viewpoint image on the display, and includes second data for displaying the specific image on the display between the completion of setting the viewpoint information and the display of the virtual viewpoint image on the display, according to the setting completion information. Image processing device.

5. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created without using the aforementioned multiple captured images and the virtual viewpoint image on a display based on the viewpoint information. The viewpoint information includes setting completion information indicating that the setting of the viewpoint information has been completed. The aforementioned data is second data for displaying the virtual viewpoint image on the display, and includes second data for displaying the specific image on the display between the completion of setting the viewpoint information and the display of the virtual viewpoint image on the display, according to the setting completion information. Image processing device.

6. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. The system outputs data for displaying the virtual viewpoint image on the display, and outputs data for displaying a specific image on the display at a timing determined according to the viewpoint information. The viewpoint information includes setting completion information indicating that the setting of the viewpoint information has been completed. The aforementioned data is second data for displaying the virtual viewpoint image on the display, and includes second data for displaying the specific image on the display between the completion of setting the viewpoint information and the display of the virtual viewpoint image on the display, according to the setting completion information. Image processing device.

7. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created in a process different from the aforementioned virtual viewpoint image and the aforementioned virtual viewpoint image on a display based on the viewpoint information. The aforementioned data is A ninth data for displaying the virtual viewpoint image on the display, wherein the ninth data is for displaying the specific image on the display at the timing when the display of the virtual viewpoint image by the display begins, or A ninth data for displaying the virtual viewpoint image on the display, which includes a ninth data for displaying the virtual viewpoint image on the display at the timing when the display of the specific image on the display has finished, The aforementioned processor, A tenth data is output to display a reception screen that accepts the aforementioned viewpoint information on the display. The data including the 9th data is output on the condition that the reception of the viewpoint information via the reception screen has been completed. Image processing device.

8. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created without using the aforementioned multiple captured images and the virtual viewpoint image on a display based on the viewpoint information. The aforementioned data is A ninth data for displaying the virtual viewpoint image on the display, wherein the ninth data is for displaying the specific image on the display at the timing when the display of the virtual viewpoint image by the display begins, or A ninth data for displaying the virtual viewpoint image on the display, which includes a ninth data for displaying the virtual viewpoint image on the display at the timing when the display of the specific image on the display has finished, The aforementioned processor, A tenth data is output to display a reception screen that accepts the aforementioned viewpoint information on the display. The data including the 9th data is output on the condition that the reception of the viewpoint information via the reception screen has been completed. Image processing device.

9. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. The system outputs data for displaying the virtual viewpoint image on the display, and outputs data for displaying a specific image on the display at a timing determined according to the viewpoint information. The aforementioned data is A ninth data for displaying the virtual viewpoint image on the display, wherein the ninth data is for displaying the specific image on the display at the timing when the display of the virtual viewpoint image by the display begins, or A ninth data for displaying the virtual viewpoint image on the display, which includes a ninth data for displaying the virtual viewpoint image on the display at the timing when the display of the specific image on the display has finished, The aforementioned processor, A tenth data is output to display a reception screen that accepts the aforementioned viewpoint information on the display. The data including the 9th data is output on the condition that the reception of the viewpoint information via the reception screen has been completed. Image processing device.

10. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created in a process different from the aforementioned virtual viewpoint image and the aforementioned virtual viewpoint image on a display based on the viewpoint information. The aforementioned processor, Further information on the point of focus that can identify the location of the point of focus is obtained, The display timing of the specific image is controlled according to the fluctuations of the gaze point identified from the gaze point information. Image processing device.

11. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. Data is output to display a specific image created without using the aforementioned multiple captured images and the virtual viewpoint image on a display based on the viewpoint information. The aforementioned processor, Further information on the point of focus that can identify the location of the point of focus is obtained, The display timing of the specific image is controlled according to the fluctuations of the gaze point identified from the gaze point information. Image processing device.

12. Processor and The processor comprises, The aforementioned processor, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information. The system outputs data for displaying the virtual viewpoint image on the display, and outputs data for displaying a specific image on the display at a timing determined according to the viewpoint information. The aforementioned processor, Further information on the point of focus that can identify the location of the point of focus is obtained, The display timing of the specific image is controlled according to the fluctuations of the gaze point identified from the gaze point information. Image processing device.

13. The aforementioned viewpoint information includes time parameters relating to time, The aforementioned data is first data for displaying the virtual viewpoint image on the display, and includes first data for displaying the specific image on the display according to the time parameter. The image processing apparatus according to any one of claims 1 to 12.

14. The aforementioned data is third data for displaying the virtual viewpoint image on the display, and includes third data for displaying the specific image on the display in accordance with the timing at which the continuity of the viewpoint information is interrupted. The image processing apparatus according to any one of claims 1 to 13.

15. The aforementioned viewpoint information includes viewpoint path information indicating a viewpoint path for observing the subject, The aforementioned data is a fourth data for displaying the virtual viewpoint image on the display, and includes a fourth data for displaying the specific image on the display at intervals defined by the viewpoint path information. The image processing apparatus according to any one of claims 1 to 14.

16. The viewpoint information includes time information indicating the time required for a first viewpoint observing the subject to move from a first position to a second position different from the first position. The aforementioned data is a fifth data for displaying the virtual viewpoint image on the display, and includes a fifth data for displaying the specific image on the display at intervals in which the required time, as indicated by the required time information, is divided. The image processing apparatus according to any one of claims 1 to 15.

17. The viewpoint information includes the position of a second viewpoint for observing the subject and elapsed time information indicating the elapsed time corresponding to the position of the second viewpoint. The aforementioned data is a sixth data for displaying the virtual viewpoint image on a display, and includes a sixth data for displaying the specific image on the display at a timing determined according to the relationship between the elapsed time indicated by the elapsed time information and the position of the second viewpoint. The image processing apparatus according to any one of claims 1 to 16.

18. The viewpoint information includes movement speed information that can identify the movement speed of the position of the third viewpoint observing the subject, The aforementioned data is a seventh data for displaying the virtual viewpoint image on the display, and includes a seventh data for displaying the specific image on the display at the timing when the movement speed identified from the movement speed information falls below a threshold. The image processing apparatus according to any one of claims 1 to 17.

19. The viewpoint information includes angle of view information relating to the angle of view for observing the subject, The aforementioned data is eighth data for displaying the virtual viewpoint image on the display, and the specific image is displayed on the display at a timing determined according to the field of view information. Includes the eighth data to be displayed during gameplay. The image processing apparatus according to any one of claims 1 to 18.

20. The aforementioned data is A ninth data for displaying the virtual viewpoint image on the display, wherein the ninth data is for displaying the specific image on the display at the timing when the display of the virtual viewpoint image by the display begins, or A ninth piece of data for displaying the virtual viewpoint image on the display, which includes a ninth piece of data for displaying the virtual viewpoint image on the display at the timing when the display of the specific image on the display has finished. The image processing apparatus according to any one of claims 1 to 3.

21. Based on multiple captured images and viewpoint information generated by capturing images of the subject by multiple imaging devices, a virtual viewpoint image is obtained that shows the appearance of the subject when observed from a viewpoint specified by the viewpoint information, and, A specific image created by a process different from the aforementioned virtual viewpoint image, and data for displaying the aforementioned virtual viewpoint image on a display based on the viewpoint information, wherein the data for displaying the specific image on the display is output at a timing determined according to the viewpoint information. Image processing methods including [specific details omitted].

22. Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information, and, Data for displaying a specific image created without using the aforementioned multiple captured images and the virtual viewpoint image on a display based on the viewpoint information, wherein data for displaying the specific image on the display is output at a timing determined according to the viewpoint information. Image processing methods including [specific details omitted].

23. Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject as it would appear when observed from a viewpoint identified by the viewpoint information. Outputting data for displaying the aforementioned virtual viewpoint image on a display, and Outputting data to display a specific image on the display at a timing determined according to the viewpoint information. Image processing methods including [specific details omitted].

24. On the computer, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information, and, A specific image created by a process different from the aforementioned virtual viewpoint image, and the virtual viewpoint image A program for executing a process that includes outputting data for displaying a specific image on a display at a timing determined according to the viewpoint information, based on the viewpoint information.

25. On the computer, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject when it is observed from a viewpoint identified by the viewpoint information, and, A program for executing a process that includes outputting data for displaying a specific image created without using the aforementioned plurality of captured images and the virtual viewpoint image on a display based on the viewpoint information, the data for displaying the specific image on the display at a timing determined according to the viewpoint information.

26. On the computer, Based on multiple captured images and viewpoint information, a virtual viewpoint image is obtained that shows the appearance of the subject as it would appear when observed from a viewpoint identified by the viewpoint information. Outputting data for displaying the aforementioned virtual viewpoint image on a display, and A program for causing a process to be executed that includes outputting data for displaying a specific image on the display at a timing determined according to the viewpoint information.

Images