Image processing apparatus, image processing method, and storage medium

Abstract

To provide an image processing apparatus capable of showing an object image to a viewer of a virtual viewpoint image.SOLUTION: The image processing apparatus includes a processor and a memory connected to or incorporated in the processor, and the processor acquires a first virtual viewpoint image generated based on a plurality of captured images, acquires viewpoint information, acquires position information of an object captured in the captured images, and acquires a second virtual viewpoint image including an object image indicating the object based on the viewpoint information and the position information.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] JP 2020-003883 A discloses an image generation device comprising: an image acquisition means for acquiring image data based on photography from multiple directions using multiple cameras that capture images of the target area; an information acquisition means for acquiring viewpoint information indicating a virtual viewpoint; and a generation means for generating a virtual viewpoint image according to the virtual viewpoint indicated by the viewpoint information based on the image data acquired by the image acquisition means and the viewpoint information acquired by the information acquisition means, the generation means generating a virtual viewpoint image according to the virtual viewpoint indicated by the viewpoint information by deforming the shape in three-dimensional space of a specific object located in the target area according to the position of the virtual viewpoint.

[0003] Japanese Patent Application Laid-Open Publication No. 2014-041259 discloses an advertising distribution device comprising: a receiving means for receiving a display request including a viewpoint condition and advertising information; an advertising space setting means for setting an advertising space according to the viewpoint condition in video data of an arbitrary viewpoint position generated based on imaging data with different viewpoint positions; an advertising information setting means for setting advertising information received by the receiving means in the advertising space set by the advertising space setting means; and a video transmission means for transmitting video data of the arbitrary viewpoint position in which advertising information has been set in the advertising space to a terminal device. Summary of the Invention

[0004] One embodiment of the technique of the present disclosure provides an image processing device, an image processing method, and a program that can show an object image to a viewer of a virtual viewpoint image. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure is an image processing device that includes a processor and a memory connected to or built into the processor, in which the processor acquires a first virtual viewpoint image generated based on a plurality of captured images, acquires viewpoint information, acquires position information of an object appearing in the captured image, and acquires a second virtual viewpoint image that includes an object image showing the object based on the viewpoint information and the position information.

[0006] A second aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which a processor acquires viewpoint information by accepting viewpoint information, and determines the accepted viewpoint information based on position information, thereby performing a first control to include an object image in a second virtual viewpoint image.

[0007] A third aspect of the technology of the present disclosure is the image processing device according to the second aspect, in which the viewpoint information includes the received first viewpoint path.

[0008] A fourth aspect of the technology disclosed herein is an image processing device according to the second aspect, in which the viewpoint information is information that identifies an area indicated by a second virtual viewpoint image, and the processor acquires the viewpoint information by accepting the viewpoint information within a range in which a position identified from the position information is included in the area.

[0009] A fifth aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which the processor changes at least one of the position information and the position of the object image when the area identified based on the viewpoint information does not include the position identified from the position information.

[0010] A sixth aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which a processor changes at least one of the position information and the position of the object image when the viewpoint information and the position information satisfy a first condition.

[0011] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to sixth aspects, in which the processor performs a second control to include the object image in the second virtual viewpoint image by moving the object image based on position information.

[0012] An eighth aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which a processor performs a third control to include an object image in a second virtual viewpoint image by changing viewpoint information based on position information.

[0013] A ninth aspect of the technology of the present disclosure is an image processing device according to the eighth aspect, in which the viewpoint information includes at least one of start point position information that can identify the position of the start point of the second viewpoint path, end point position information that can identify the position of the end point of the second viewpoint path, first gaze direction information that can identify the first gaze direction, and field of view information that can identify the field of view.

[0014] A tenth aspect of the technology of the present disclosure is an image processing device according to the eighth or ninth aspect, in which the viewpoint information includes second gaze direction information capable of identifying a second gaze direction, and the third control includes control to include an object image in the second virtual viewpoint image by changing the second gaze direction information based on position information of at least one of the position of the start point of the third viewpoint path and the position of the end point of the third viewpoint path as the viewpoint information.

[0015] An eleventh aspect of the technology of the present disclosure is an image processing device according to any one of the second to fourth aspects, in which the second virtual viewpoint image includes a first object image showing a object, and the processor performs first control within a range in which at least one of the size and position of the first object image within the second virtual viewpoint image satisfies a second condition.

[0016] A twelfth aspect of the technology of the present disclosure is an image processing device according to the seventh aspect, in which the processor performs second control based on at least one of the size and position of a second subject image showing a subject in a third virtual viewpoint image generated based on viewpoint information.

[0017] A thirteenth aspect of the technology of the present disclosure is an image processing device according to any one of the eighth to tenth aspects, in which a processor performs a third control based on at least one of the size and position of a third subject image showing a subject in a third virtual viewpoint image generated based on viewpoint information.

[0018] A fourteenth aspect of the technology of the present disclosure is an image processing device according to the second, third, fourth, or eleventh aspect, in which objects are assigned display priorities, and when multiple objects with assigned priorities appear in a captured image, the processor performs a first control based on the priorities.

[0019] A fifteenth aspect of the technology of the present disclosure is an image processing device according to the seventh or twelfth aspect, in which objects are assigned display priorities, and when multiple objects with assigned priorities appear in a captured image, the processor performs a second control based on the priorities.

[0020] A sixteenth aspect of the technology of the present disclosure is an image processing device according to the eighth, ninth, tenth, or thirteenth aspect, in which objects are assigned display priorities, and when multiple objects with assigned priorities appear in a captured image, the processor performs a third control based on the priorities.

[0021] A seventeenth aspect of the technique of the present disclosure is the image processing device according to any one of the fourteenth to sixteenth aspects, in which the priority is determined based on an attribute of the object.

[0022] An 18th aspect of the technology of the present disclosure is an image processing device according to any one of the 14th to 17th aspects, in which a processor determines a priority based on the attributes of a user who set the viewpoint information.

[0023] A 19th aspect of the technology of the present disclosure is an image processing device according to any one of the 14th to 18th aspects, in which a processor determines a priority based on the status of an object to be imaged by multiple imaging devices.

[0024] A twentieth aspect of the technology of the present disclosure is an image processing device according to any one of the first to nineteenth aspects, in which a processor changes the display mode of an object image based on viewpoint information and position information.

[0025] A 21st aspect of the technology of the present disclosure is an image processing device according to any one of the first to 20th aspects, in which a processor outputs data for displaying a second virtual viewpoint image on a display for a period of time determined according to viewpoint information.

[0026] A 22nd aspect of the technology of the present disclosure is an image processing method that includes acquiring a first virtual viewpoint image generated based on a plurality of captured images, acquiring viewpoint information, acquiring position information of an object appearing in the captured images, and acquiring a second virtual viewpoint image that includes an object image showing the object based on the viewpoint information and the position information.

[0027] A 23rd aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a first virtual viewpoint image generated based on a plurality of captured images, acquiring viewpoint information, acquiring position information of an object depicted in the captured image, and acquiring a second virtual viewpoint image including an object image showing the object based on the viewpoint information and position information. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of an image processing system. [Figure 2] 1 is a conceptual diagram showing an example of the schematic configuration of a soccer stadium and an example of the schematic configuration of a rectangular parallelepiped that defines three-dimensional coordinates that can identify a position within the soccer stadium. [Figure 3] FIG. 2 is a block diagram illustrating an example of the hardware configuration of an electrical system of a user device. [Figure 4] FIG. 2 is a block diagram showing an example of main functions of a CPU of the image processing device. [Figure 5] 10A and 10B are conceptual diagrams illustrating an example of processing content of a reception screen generating unit and an example of display content on a display of a user device. [Figure 6] 10 is a screen diagram showing an example of a display mode of a reception screen when the operation mode of the user device is a viewpoint setting mode. FIG. [Figure 7] 10 is a screen diagram showing an example of a display mode of a reception screen when the operation mode of the user device is a gaze point setting mode. FIG. [Figure 8] 10 is a block diagram showing an example of the contents of viewpoint information and an example of a manner in which viewpoint information is transmitted from a user device to an image processing device. FIG. [Figure 9] FIG. 10 is a conceptual diagram illustrating an example of processing content of a virtual viewpoint image generating unit. [Figure 10] FIG. 10 is a conceptual diagram showing an example of a viewpoint from which an advertising billboard can be seen and a viewpoint from which an advertising billboard cannot be seen. [Figure 11] FIG. 10 is a conceptual diagram illustrating an example of processing content of a virtual viewpoint image generating unit. [Figure 12] FIG. 10 is a conceptual diagram showing an example of a manner in which a virtual viewpoint video is transmitted from an image processing device to a user device. [Figure 13] 10A and 10B are screen diagrams showing an example of a state in which a virtual viewpoint video and a reception screen are displayed on the display of a user device. [Figure 14] 10 is a flowchart illustrating an example of the flow of an image generation process. [Figure 15] FIG. 10 is a conceptual diagram showing a second modified example. [Figure 16] FIG. 10 is a conceptual diagram showing a third modified example. [Figure 17] FIG. 10 is a conceptual diagram showing a fourth modified example. [Figure 18] FIG. 10 is a conceptual diagram showing a fifth modified example. [Figure 19] FIG. 10 is a conceptual diagram showing a sixth modified example. [Figure 20] FIG. 13 is a conceptual diagram showing a seventh modified example. [Figure 21] FIG. 13 is a conceptual diagram showing a seventh modified example. [Figure 22] FIG. 13 is a conceptual diagram showing an eighth modified example. [Figure 23] FIG. 13 is a conceptual diagram showing a ninth modified example. [Figure 24] FIG. 10 is a conceptual diagram showing an example of how an image generation processing program stored in a storage medium is installed in a computer of an image processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0029] An example of an embodiment of an image processing device, an image processing method, and a program according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor processing unit". NVM is an abbreviation for "Non-Volatile Memory". RAM is an abbreviation for "Random Access Memory". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". LAN is an abbreviation for "Local Area Network". USB is an abbreviation for "Universal Serial Bus". HMD is an abbreviation for "Head Mounted Display". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)". TDM is an abbreviation for "Time-Division Multiplexing". GNSS is an abbreviation for "Global Navigation Satellite System". AI is an abbreviation for "Artificial Intelligence".

[0032] As an example, as shown in FIG. 1, an image processing system 2 includes an image processing device 10 and a user device 12.

[0033] In this embodiment, a server is used as an example of the image processing device 10. The server is realized by, for example, a mainframe. However, this is merely an example, and the server may be realized by network computing such as cloud computing, fog computing, edge computing, or grid computing. Furthermore, the image processing device 10 may be a personal computer, multiple personal computers, multiple servers, or a device that combines a personal computer and a server.

[0034] In this embodiment, a smartphone is used as an example of the user device 12. However, the smartphone is merely an example, and the user device 12 may be, for example, a personal computer, or a portable multi-function terminal such as a tablet terminal or a head-mounted display (hereinafter referred to as "HMD").

[0035] In this embodiment, the image processing device 10 and the user device 12 are communicatively connected via, for example, a base station (not shown). The communication standards used in the base station include wireless communication standards such as the 5G standard and / or the LTE standard, wireless communication standards such as the WiFi (802.11) standard and / or the Bluetooth (registered trademark) standard, and wired communication standards such as the TDM standard and / or the Ethernet (registered trademark) standard.

[0036] The image processing device 10 acquires an image and transmits the acquired image to the user device 12. Here, the image refers to, for example, a captured image 64 (see FIG. 5, etc.) obtained by capturing an image, an image generated based on the captured image 64 (see FIG. 5, etc.), etc. An example of an image generated based on the captured image (see FIG. 5, etc.) is a virtual viewpoint image 76 (see FIG. 11, etc.).

[0037] The user device 12 is used by a user 14. The user device 12 includes a touch panel display 16. The touch panel display 16 is realized by a display 18 and a touch panel 20. An example of the display 18 is an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0038] The touch panel display 16 is formed by overlaying the touch panel 20 on the display area of ​​the display 18, or by using an in-cell type in which the touch panel function is built into the display 18. Note that the in-cell type is merely an example, and the display may also be an out-cell type or an on-cell type.

[0039] The user device 12 executes processing in accordance with instructions received from a user via the touch panel 20 or the like. For example, the user device 12 exchanges various types of information with the image processing device 10 in accordance with instructions received from a user via the touch panel 20 or the like.

[0040] The user device 12 receives the image transmitted from the image processing device 10 and displays the received image on the display 18. The user 14 views the image displayed on the display 18.

[0041] The image processing device 10 includes a computer 22, a transmission / reception device 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure, and includes a CPU 28, an NVM 30, and a RAM 32. The image processing device 10 includes a bus 34, and the CPU 28, the NVM 30, and the RAM 32 are connected via the bus 34. In the example shown in FIG. 1, for convenience of illustration, a single bus is shown as the bus 34, but multiple buses may be used. The bus 34 may also include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, etc.

[0042] The CPU 28 is an example of a "processor" according to the technology of the present disclosure. The CPU 28 controls the entire image processing device 10. The NVM 30 stores various parameters, various programs, etc. Examples of the NVM 30 include an EEPROM, an SSD, and / or an HDD. The RAM 32 is an example of a "memory" according to the technology of the present disclosure. Various pieces of information are temporarily stored in the RAM 32. The RAM 32 is used as a work memory by the CPU 28.

[0043] The transceiver 24 is connected to the bus 34. The transceiver 24 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the user device 12 via a base station (not shown) under the control of the CPU 28. In other words, the CPU 28 transmits and receives various types of information to and from the user device 12 via the transceiver 24.

[0044] The communication I / F 26 is realized by, for example, a device having an FPGA. The communication I / F 26 is connected to a plurality of image capture devices 36 via LAN cables (not shown). The image capture devices 36 are image capture devices having CMOS image sensors and equipped with optical zoom functions and / or digital zoom functions. Note that other types of image sensors, such as CCD image sensors, may be used instead of the CMOS image sensors.

[0045] For example, multiple imaging devices 36 are installed in a soccer stadium 4 (see FIG. 2) and capture images of subjects within the soccer stadium 4. Captured images 64 (see FIG. 5, etc.) obtained by capturing images of subjects with the imaging devices 36 are used to generate, for example, a virtual viewpoint image 76 (see FIG. 11, etc.). Therefore, each of the multiple imaging devices 36 is installed at a different location within the soccer stadium, i.e., at a location where multiple captured images 64 (see FIG. 5, etc.) from which the virtual viewpoint image 76 (see FIG. 11, etc.) can be generated are obtained. The soccer stadium 4 is a three-dimensional area including a soccer field 4A (see FIG. 2) and spectator stands 4B (see FIG. 2) constructed to surround the soccer field 4A, and is observed by a user 14. The observer, i.e., the user 14, can observe the inside of the soccer stadium 4 from the spectator stands or a location outside the soccer stadium 4 through an image displayed on the display 18 of the user device 12.

[0046] Here, a soccer stadium is used as an example of a location where multiple imaging devices 36 are installed, but the technology of the present disclosure is not limited to this, and the location where multiple imaging devices 36 are installed may be any location where multiple imaging devices 36 can be installed, such as a baseball stadium, rugby field, curling court, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater venue.

[0047] The communication I / F 26 is connected to the bus 34 and controls the exchange of various information between the CPU 28 and the multiple imaging devices 36. For example, the communication I / F 26 controls the multiple imaging devices 36 in accordance with a request from the CPU 28. The communication I / F 26 outputs, to the CPU 28, captured images 64 (see FIGS. 3, 4, 8, etc.) obtained by capturing images by each of the multiple imaging devices 36. Note that, although the communication I / F 26 is exemplified here as a wired communication I / F, it may also be a wireless communication I / F such as a high-speed wireless LAN.

[0048] The NVM 30 stores an image generation processing program 38. The image generation processing program 38 is an example of a “program” according to the technology of the present disclosure. The CPU 28 reads the image generation processing program 38 from the NVM 30 and executes the image generation processing program 38 on the RAM 32 to perform image generation processing (see FIG. 14 ).

[0049] As an example, as shown in Fig. 2, multiple advertising billboards 6 are installed around a soccer field 4A. For example, information about advertisers is written on the advertising billboards 6 as letters, pictures, and / or patterns. The advertising billboards 6 are an example of an "object" and a "subject" according to the technology of the present disclosure.

[0050] The soccer stadium 4 is assigned coordinates that can identify its position within the soccer stadium 4. Here, as an example of coordinates that can identify its position within the soccer stadium 4, three-dimensional coordinates that can identify its position within a virtual rectangular parallelepiped 8 that contains the soccer stadium 4, with one vertex of the rectangular parallelepiped 8 as the origin, are applied.

[0051] The multiple advertising billboards 6 are assigned location information 39. An example of the location information 39 is three-dimensional coordinates that can identify the size and location of the advertising billboard 6 within the soccer stadium 4. However, this is merely an example, and the location information 39 may also be information indicating latitude, longitude, and altitude obtained using GNSS.

[0052] Furthermore, the installation positions of the multiple imaging devices 36 (see FIG. 1) within the soccer stadium 4 can also be identified by three-dimensional coordinates, and three-dimensional coordinates are assigned to each imaging device 36.

[0053] 3, the user device 12 includes a display 18, a computer 40, an imaging device 42, a transceiver device 44, a speaker 46, a microphone 48, and an accepting device 50. The computer 40 includes a CPU 52, an NVM 54, and a RAM 56. The user device 12 includes a bus 58, and the CPU 52, the NVM 54, and the RAM 56 are connected via the bus 58.

[0054] 3, for convenience of illustration, one bus is shown as bus 58, but multiple buses may be used. Bus 58 may also include a serial bus or a parallel bus configured with a data bus, an address bus, a control bus, etc.

[0055] The CPU 52 controls the entire user device 12. The NVM 54 stores various parameters, various programs, etc. An example of the NVM 54 is an EEPROM. The RAM 56 temporarily stores various information. The RAM 56 is used as a work memory by the CPU 52. The CPU 52 reads various programs from the NVM 54 and executes the programs on the RAM 56 to perform processing in accordance with the programs.

[0056] The imaging device 42 is an imaging device having a CMOS image sensor and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The imaging device 42 is connected to a bus 58, and the CPU 52 controls the imaging device 42. The captured image obtained by imaging the imaging device 42 is acquired by the CPU 52 via the bus 58.

[0057] The transmitting / receiving device 44 is connected to the bus 58. The transmitting / receiving device 44 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the image processing device 10 via a base station (not shown) under the control of the CPU 52. That is, the CPU 52 transmits and receives various types of information to and from the image processing device 10 via the transmitting / receiving device 44.

[0058] The speaker 46 converts the electrical signal into sound. The speaker 46 is connected to the bus 58. The speaker 46 receives the electrical signal output from the CPU 52 via the bus 58, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the user device 12.

[0059] The microphone 48 converts the collected sound into an electrical signal. The microphone 48 is connected to a bus 58. The electrical signal obtained by converting the sound collected by the microphone 48 is acquired by the CPU 52 via the bus 58.

[0060] The receiving device 50 receives instructions from the user 14, etc. Examples of the receiving device 50 include a touch panel 20 and hard keys (not shown). The receiving device 50 is connected to a bus 58, and the instructions received by the receiving device 50 are acquired by a CPU 52.

[0061] As an example, as shown in FIG. 4, the CPU 28 reads the image generation processing program 38 from the NVM 30 and executes the image generation processing program 38 on the RAM 32, thereby operating as a reception screen generation unit 28A, a viewpoint information acquisition unit 28B, and a virtual viewpoint image generation unit 28C.

[0062] 5, a reception screen 66 and a virtual viewpoint video screen 68 are displayed on the display 18 of the user device 12. In the example shown in Fig. 5, the reception screen 66 and the virtual viewpoint video screen 68 are displayed side by side on the display 18. Note that this is merely an example, and the reception screen 66 and the virtual viewpoint video screen 68 may be switched and displayed in accordance with an instruction given by the user 14 to the touch panel 20, or the reception screen 66 and the virtual viewpoint video screen 68 may be displayed separately on different display devices.

[0063] Furthermore, in this embodiment, the reception screen 66 is displayed on the display 18 of the user device 12, but the technology of the present disclosure is not limited to this, and for example, the reception screen 66 may be displayed on a display connected to a device (e.g., a personal computer) used by a person who creates or edits the virtual viewpoint image 78 (see FIG. 11).

[0064] The user device 12 communicates with the image processing device 10 to acquire a virtual viewpoint video 78 (see FIG. 11) from the image processing device 10. The virtual viewpoint video 78 (see FIG. 11) acquired by the user device 12 from the image processing device 10 is displayed on a virtual viewpoint video screen 68 of the display 18. In the example shown in FIG. 5, the virtual viewpoint video 78 is not displayed on the virtual viewpoint video screen 68.

[0065] The user device 12 communicates with the image processing device 10 to acquire reception screen data 70 representing a reception screen 66 from the image processing device 10. The display 18 displays the reception screen 66 represented by the reception screen data 70 acquired by the user device 12 from the image processing device 10.

[0066] The reception screen 66 includes an overhead view video screen 66A, a guidance message display area 66B, a decision key 66C, and a cancel key 66D, and displays various information necessary for generating a virtual viewpoint video 78 (see FIG. 8). The user 14 refers to the reception screen 66 and gives instructions to the user device 12. The instructions from the user 14 are received by, for example, the touch panel 20.

[0067] The overhead video screen 66A displays an overhead video 72. The overhead video 72 is a moving image showing the state of the soccer stadium when observed from above, and is generated based on a plurality of captured images 64 obtained by capturing images by at least one of the plurality of imaging devices 36. An example of the overhead video 72 is a live broadcast video.

[0068] The guidance message display area 66B displays various messages indicating the contents of operations required of the user 14. The operations required of the user 14 refer to, for example, operations required to generate the virtual viewpoint video 78 (see FIG. 11) (for example, an operation to set a viewpoint, an operation to set a point of interest, etc.).

[0069] The display content of the guidance message display area 66B changes depending on the operation mode of the user device 12. For example, the user device 12 has, as operation modes, a viewpoint setting mode for setting a viewpoint and a gaze point setting mode for setting a gaze point, and the display content of the guidance message display area 66B differs between the viewpoint setting mode and the gaze point setting mode.

[0070] The enter key 66C and the cancel key 66D are both soft keys. The enter key 66C is pressed by the user 14 when deciding on an instruction accepted on the reception screen 66. The cancel key 66D is pressed by the user 14 when canceling an instruction accepted on the reception screen 66.

[0071] The reception screen generation unit 28A acquires a plurality of captured images 64 from a plurality of imaging devices 36. The captured images 64 include imaging condition information 64A. The imaging condition information 64A refers to information indicating the imaging conditions. Examples of the imaging conditions include three-dimensional coordinates that can identify the installation position of the imaging device 36, the imaging direction of the imaging device 36, the angle of view used in imaging by the imaging device 36, and the zoom magnification applied to the imaging device 36.

[0072] The reception screen generation unit 28A generates an overhead video 72 based on a plurality of captured images 64 acquired from a plurality of imaging devices 36. Then, the reception screen generation unit 28A generates data showing a reception screen 66 including the overhead video 72 as reception screen data 70.

[0073] The reception screen generation unit 28A outputs the reception screen data 70 to the transmission / reception device 24. The transmission / reception device 24 transmits the reception screen data 70 input from the reception screen generation unit 28A to the user device 12. The user device 12 receives the reception screen data 70 transmitted from the transmission / reception device 24 at the transmission / reception device 44 (see FIG. 3 ). The reception screen 66 indicated by the reception screen data 70 received by the transmission / reception device 44 is displayed on the display 18.

[0074] 6, when the operation mode of the user device 12 is the viewpoint setting mode, a message 66B1 is displayed in the guidance message display area 66B of the reception screen 66. The message 66B1 is a message that prompts the user 14 to specify the viewpoint to be used for generating the virtual viewpoint video 78 (see FIG. 11). Here, the viewpoint refers to a virtual viewpoint from which the inside of the soccer stadium 4 is observed. The virtual viewpoint refers to the position where a virtual camera that captures an image of a subject is installed, rather than the position where an actual camera such as a physical camera (e.g., the imaging device 36) that captures an image of the subject is installed.

[0075] Touch panel 20 accepts an instruction from user 14 while message 66B1 is displayed in guidance message display area 66B. In this case, the instruction from user 14 refers to an instruction of a viewpoint. The viewpoint corresponds to the position of a pixel in overhead view image 72. The position of a pixel in overhead view image 72 corresponds to a position in soccer stadium 4 (see FIG. 2). The viewpoint is specified by user 14 specifying the position of a pixel in overhead view image 72 via touch panel 20. Note that the viewpoint may be three-dimensional coordinates corresponding to a three-dimensional position in overhead view image 72. Any method can be used to specify the three-dimensional position. For example, user 14 may directly input the three-dimensional coordinate position, or the three-dimensional coordinate position may be specified by displaying two images of soccer stadium 4 viewed from two planes perpendicular to each other and specifying the pixel position in each image.

[0076] 6, a viewpoint path P1, which is a path for observing a subject, is shown as an example of a viewpoint. The viewpoint path P1 is an example of a "first viewpoint path," a "second viewpoint path," and a "third viewpoint path" according to the technology of the present disclosure.

[0077] The viewpoint path P1 is a collection of multiple viewpoints arranged in a line from a start point P1s to an end point P1e. ​​The viewpoint path P1 is defined along a path (a meandering path from the start point P1s to the end point P1e in the example shown in FIG. 6) along which the user 14 slides (swipes) their fingertip 14A over an area of ​​the entire area of ​​the touch panel 20 that corresponds to the display area of ​​the overhead-view image 72. The observation time from the viewpoint path P1 (e.g., the time spent observing between two different viewpoints and / or the time spent stationary at one point) is defined by the speed of the slide made on the touch panel 20 when forming the viewpoint path P1 via the touch panel 20, the time spent staying at one viewpoint on the viewpoint path P1 (e.g., the time spent long-pressing), and the like.

[0078] In the example shown in FIG. 6, when the viewpoint path P1 is to be confirmed, the enter key 66C is turned on, and when the viewpoint path P1 is to be cancelled, the cancel key 66D is turned on.

[0079] 6, only viewpoint path P1 is set, but this is merely an example, and multiple viewpoint paths may be set. Also, instead of being limited to viewpoint paths, multiple discontinuous viewpoints or a single viewpoint may be set.

[0080] As an example, as shown in FIG. 7, when the operation mode of the user device 12 is the gaze point setting mode, a message 66B2 is displayed in the guidance message display area 66B of the reception screen 66. The message 66B2 is a message that prompts the user 14 to specify a gaze point to be used in generating a virtual viewpoint image 78 (see FIG. 11). Here, the gaze point refers to a point that is virtually gazed upon when observing the inside of a soccer stadium from a viewpoint. Once the viewpoint and gaze point are set, the virtual gaze direction (imaging direction of the virtual camera) is also uniquely determined. The virtual gaze direction refers to the direction from the viewpoint toward the gaze point.

[0081] Touch panel 20 accepts an instruction from user 14 while message 66B2 is displayed in guidance message display area 66B. In this case, the instruction from user 14 refers to an instruction of a point of gaze. The point of gaze corresponds to the position of a pixel in overhead view image 72. The position of the pixel in overhead view image 72 corresponds to a position in soccer stadium 4 (see FIG. 2 ). The instruction of the point of gaze is made by user 14 indicating the position of a pixel in overhead view image 72 via touch panel 20. In the example shown in FIG. 7, point of gaze GP is shown. Point of gaze GP is defined according to the location where user 14 touches with fingertip 14A on an area of ​​touch panel 20 that corresponds to the display area of ​​overhead view image 72. In the example shown in FIG. 7, to confirm point of gaze GP, enter key 66C is pressed, and to cancel point of gaze GP, cancel key 66D is pressed. Note that the point of gaze may be a three-dimensional coordinate corresponding to a three-dimensional position in overhead view image 72. As with the method for specifying the viewpoint position, any method can be used to specify the three-dimensional position.

[0082] 7, only the gaze point GP is specified, but this is merely an example, and there may be multiple gaze points, or a path (gazing point path) in which multiple gaze points are arranged in a line. The gaze point path may be one or multiple.

[0083] 8, the CPU 52 of the user device 12 generates viewpoint information 74 based on a viewpoint path P1 and a gaze point GP. The viewpoint information 74 is information used to generate a virtual viewpoint video 78 (see FIG. 11). The viewpoint information 74 includes viewpoint path information 74A, line-of-sight direction information 74B, required time information 74C, elapsed time information 74D, movement speed information 74E, and angle-of-view information 74F.

[0084] The viewpoint path information 74A is information that can identify the position of a pixel in the overhead image 72 of the viewpoint path P1 (see FIG. 6) that was confirmed in the viewpoint setting mode. An example of information that can identify the position of a pixel in the overhead image 72 of the viewpoint path P1 is coordinates that can identify the position of a pixel of the viewpoint path P1 in the overhead image 72. The viewpoint path P1 includes a start point P1s and an end point P1e (see FIG. 6). Therefore, the viewpoint path information 74A also includes start point position information (hereinafter also simply referred to as "start point position information") that can identify the position of the start point P1s, and end point position information (hereinafter also simply referred to as "end point position information") that can identify the position of the end point P1e. ​​An example of the start point position information is coordinates that can identify the position of the pixel of the start point P1s in the overhead image 72. An example of the end point position information is coordinates that can identify the position of the pixel of the end point P1e in the overhead image 72.

[0085] The gaze direction information 74B is information that can identify the gaze direction. The gaze direction refers to, for example, the direction in which a subject is observed from the viewpoint path P1 toward the gaze point GP. The gaze direction information 74B is determined for each viewpoint included in the viewpoint path information 74A, and is specified by information that can identify the position of the viewpoint (for example, coordinates that can identify the pixel position of the viewpoint in the overhead image 72) and information that can identify the position of the gaze point GP established in the gaze point setting mode (for example, coordinates that can identify the pixel position of the gaze point GP in the overhead image 72). Note that the gaze direction here is an example of a "first gaze direction" and a "second gaze direction" according to the technology of the present disclosure, and the gaze direction information 74B is an example of "first gaze direction information" and "second gaze direction information" according to the technology of the present disclosure.

[0086] The required time information 74C is information indicating the required time (hereinafter also simply referred to as "required time") required for the viewpoint for observing the subject on the viewpoint path P1 to move from a first position to a second position different from the first position. Here, the first position refers to the start point P1s (see FIGS. 6 and 7), and the second position refers to, for example, the end point P1e (see FIGS. 6 and 7). Note that this is merely an example, and the first position may be the start point P1s (see FIGS. 6 and 7) and the second position may be the position of a viewpoint at an intermediate point on the viewpoint path P1, or the first position may be the position of a viewpoint at an intermediate point on the viewpoint path P1 and the second position may be the end point P1e (see FIGS. 6 and 7).

[0087] The elapsed time information 74D is information indicating the position of the viewpoint from which the subject is observed on the viewpoint path P1 and the elapsed time corresponding to the viewpoint position. The elapsed time corresponding to the viewpoint position (hereinafter simply referred to as "elapsed time") refers to, for example, the time during which the viewpoint remains stationary at a certain viewpoint position on the viewpoint path P1.

[0088] The movement speed information 74E is information that can identify the movement speed of the position of the viewpoint from which the subject is observed on the viewpoint path P1. The movement speed of the viewpoint position (hereinafter also simply referred to as "movement speed") refers to, for example, the speed of sliding performed on the touch panel 20 when forming the viewpoint path P1 via the touch panel 20. The movement speed information 74E is associated with each viewpoint in the viewpoint path P1.

[0089] The angle of view information 74F is information that can identify the angle of view (hereinafter, also simply referred to as the "angle of view"). Here, the angle of view refers to the viewing angle at which the subject is observed on the viewpoint path P1. In this embodiment, the angle of view is fixed to a predetermined angle (e.g., 100 degrees). However, this is merely an example, and the angle of view may be determined according to the moving speed. In this case, for example, within a range in which an upper limit (e.g., 150 degrees) and a lower limit (e.g., 15 degrees) of the angle of view are set, the angle of view narrows as the moving speed decreases. Alternatively, the angle of view may be narrowed as the moving speed increases. Alternatively, the angle of view may be determined according to the elapsed time. In this case, for example, the angle of view may be minimized when the elapsed time exceeds a first predetermined time (e.g., 3 seconds), or maximized when the elapsed time exceeds the first predetermined time. Alternatively, the angle of view may be determined according to an instruction received by the receiving device 50. In this case, the receiving device 50 may receive instructions regarding the position of the viewpoint on the viewpoint path P1 at which the angle of view is to be changed and the angle of view after the change.

[0090] The CPU 52 outputs the viewpoint information 74 to the transmission / reception device 44. The transmission / reception device 44 transmits the viewpoint information 74 input from the CPU 52 to the image processing device 10. The transmission / reception device 24 of the image processing device 10 receives the viewpoint information 74.

[0091] The viewpoint information acquisition unit 28B of the image processing device 10 acquires viewpoint information. In the example shown in Fig. 8, the viewpoint information acquisition unit 28B acquires the viewpoint information 74 by accepting the viewpoint information 74 received by the transmission / reception device 24. That is, the transmission / reception device 24 outputs the received viewpoint information 74 to the viewpoint information acquisition unit 28B, and the viewpoint information acquisition unit 28B acquires the viewpoint information 74 by inputting the viewpoint information 74 from the transmission / reception device 24.

[0092] 9, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from a plurality of imaging devices 36 in accordance with viewpoint information 74. That is, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from a plurality of imaging devices 36 to be used for generating a virtual viewpoint image 76 (see FIG. 11 ), which is an image showing the state of the subject when the subject is observed in accordance with the viewpoint information 74.

[0093] The virtual viewpoint image generation unit 28C acquires, from the NVM 30, position information 39 of the advertising billboard 6 (see FIG. 2) that appears in the captured image 64. More specifically, the virtual viewpoint image generation unit 28C references the imaging condition information 64A included in each captured image 64 and acquires, from the NVM 30, position information 39 related to the advertising billboard 6 that appears in its entirety in the captured image 64. Note that, although an example is given here in which the position information 39 related to the advertising billboard 6 is acquired from the NVM 30 on the condition that the entire advertising billboard 6 appears in the captured image 64, this is merely an example, and the position information 39 related to the advertising billboard 6 may also be acquired from the NVM 30 on the condition that a portion (e.g., 50% or more) of the advertising billboard 6 appears in the captured image 64.

[0094] The virtual viewpoint image generation unit 28C acquires a virtual viewpoint image 76 (see FIG. 11) generated based on the plurality of captured images 64. Here, the "virtual viewpoint image 76" is an example of a "first virtual viewpoint image" according to the technology of the present disclosure.

[0095] Furthermore, the virtual viewpoint image generation unit 28C acquires a virtual viewpoint image 76 including an advertising billboard image 77 (see FIG. 11 ) showing the advertising billboard 6, based on the viewpoint information 74 and the position information 39. Here, the virtual viewpoint image 76 including the advertising billboard image 77 is an example of a "second virtual viewpoint image" and a "third virtual viewpoint image" according to the technology of the present disclosure. The advertising billboard image 77 is an example of an "object image," a "first subject image," a "second subject image," and a "third subject image" according to the technology of the present disclosure.

[0096] Here, the virtual viewpoint image generation unit 28C performs first control (hereinafter also simply referred to as "first control") to include the advertising billboard image 77 in the virtual viewpoint image 76 by determining the viewpoint information 74 received by the viewpoint information acquisition unit 28B based on the position information 39. One example of determining the viewpoint information 74 is to restrict the viewpoint path information 74A. That is, the virtual viewpoint image generation unit 28C restricts the viewpoint path information 74A so as to shorten the viewpoint path P1 to a range in which the advertising billboard 6 appears in the virtual viewpoint image 76. The example shown in FIG. 9 shows a viewpoint path P2 obtained by shortening the viewpoint path P1 to a range in which the advertising billboard 6 appears in the virtual viewpoint image 76. The viewpoint path P2 includes a start point P2s and an end point P2e. The start point P2s is a viewpoint at the same position as the start point P1s, and the end point P2e is a viewpoint at a position midway along the viewpoint path P1.

[0097] In this way, when viewpoint path P1 is changed to viewpoint path P2, for example, as shown in Fig. 10, among the multiple viewpoints included in viewpoint path P2, viewpoints from which the advertising billboard 6 can be seen are adopted, and viewpoints from which the advertising billboard 6 cannot be seen are not adopted. In other words, viewpoint information 74 from which a virtual viewpoint image 76 including an advertising billboard image 77 (Fig. 11) can be generated is created by virtual viewpoint image generator 28C.

[0098] When the viewpoint path P1 is changed to the viewpoint path P2, viewpoint path information 74A indicating the viewpoint path P2 may be transmitted from the virtual viewpoint image generation unit 28C to the user device 12. In this case, the viewpoint path information 74A may be received by the user device 12, and the viewpoint path P2 indicated by the received viewpoint path information 74A may be displayed on the overhead video screen 66A. By looking at the viewpoint path P2 displayed on the overhead video screen 66A, the user 14 can visually recognize that the viewpoint that he or she originally specified has been restricted.

[0099] Also, here, the viewpoint path information 74A is restricted by shortening the length of the viewpoint path P1, but the technology of the present disclosure is not limited to this, and for example, the viewpoint path information 74A may be restricted so as to fix the direction of the viewpoint path P1 in one direction, or the viewpoint path information 74A may be restricted so as to fix the position of the start point P1s, the position of the end point P1e, and / or the position of the midpoint of the viewpoint path P1.

[0100] 11, the virtual viewpoint image generation unit 28C selects a plurality of captured images 64 in accordance with viewpoint information 74 (for example, viewpoint information 74 including viewpoint path information 74A indicating viewpoint path P2 shown in FIG. 9) determined based on the position information 39. Then, based on the selected plurality of captured images 64, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76 which is an image showing the state of the subject when the subject is observed from a viewpoint specified by the viewpoint information 74 (for example, viewpoint information 74 including viewpoint path information 74A indicating viewpoint path P2 shown in FIG. 9).

[0101] For example, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76 of multiple frames according to the viewpoint path P2 (see FIG. 9). That is, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76 for each viewpoint on the viewpoint path P2. The virtual viewpoint image 76 of multiple frames includes an advertising billboard image 77. The virtual viewpoint image generation unit 28C arranges the virtual viewpoint images 76 of multiple frames in chronological order to generate a virtual viewpoint video 78 including the advertising billboard image 77. The virtual viewpoint video 78 generated in this manner is data to be displayed on the display 18 of the user device 12 for a time period determined according to viewpoint information 74 (for example, viewpoint information 74 including viewpoint path information 74A indicating the viewpoint path P2 shown in FIG. 9).

[0102] 12 , the virtual viewpoint image generation unit 28C outputs a virtual viewpoint video 78 to the transmission / reception device 24. The transmission / reception device 24 transmits the virtual viewpoint video 78 input from the virtual viewpoint image generation unit 28C to the user device 12.

[0103] As an example, as shown in FIG. 13, the user device 12 receives a virtual viewpoint video 78 (see FIG. 3) at the transmission / reception device 44. The virtual viewpoint video 78 is a moving image in which a plurality of frames of virtual viewpoint images 76 are arranged in time series. The plurality of frames of the virtual viewpoint images 76 are displayed on the virtual viewpoint video screen 68 of the display 18 of the user device 12 at a predetermined frame rate (e.g., several tens of frames per second) from the first frame to the last frame. The virtual viewpoint video 78 is displayed on the virtual viewpoint video screen 68 for a time period according to the viewpoint information 74 (e.g., viewpoint information 74 including viewpoint path information 74A indicating viewpoint path P2 shown in FIG. 9). As a result, the virtual viewpoint video 78 including the advertising billboard image 77 is viewed by the user 14.

[0104] Fig. 14 shows an example of the flow of image generation processing performed by the CPU 28 of the image processing device 10. The flow of image generation processing shown in Fig. 14 is an example of the "image processing method" according to the technique of the present disclosure.

[0105] 14, first, in step ST10, the reception screen generating unit 28A generates reception screen data 70 based on a plurality of captured images 64 (see FIG. 5). After the processing of step ST10 is executed, the image generation processing proceeds to step ST12.

[0106] In step ST12, the reception screen generating unit 28A causes the transmission / reception device 24 to transmit the reception screen data 70 generated in step ST10 to the user device 12. After the processing of step ST12 is executed, the image generation processing proceeds to step ST14.

[0107] When the reception screen data 70 is transmitted from the image processing device 10 to the user device 12 by executing the processing of step ST12, the user device 12 receives the reception screen data 70 and displays the reception screen 66 indicated by the received reception screen data 70 on the display 18 (see FIGS. 5 to 7). When the reception screen 66 is displayed on the display 18 of the user device 12, the user 14 gives instructions such as a viewpoint and a gaze point to the user device 12 via the touch panel 20 (see FIGS. 6 and 7). The CPU 52 of the user device 12 generates viewpoint information 74 based on the viewpoint and the gaze point received by the touch panel 20, and transmits the generated viewpoint information 74 to the image processing device 10 via the transmission / reception device 44 (see FIG. 8).

[0108] In step ST14, the viewpoint information acquisition unit 28B determines whether or not the viewpoint information 74 has been received by the transmission / reception device 24. If the viewpoint information 74 has not been received by the transmission / reception device 24 in step ST14, the determination is negative, and the image generation processing proceeds to step ST26. If the viewpoint information 74 has been received by the transmission / reception device 24 in step ST14, the determination is positive, and the image generation processing proceeds to step ST16. The viewpoint information acquisition unit 28B acquires the viewpoint information 74 received by the transmission / reception device 24 (see FIG. 9).

[0109] In step ST16, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from the plurality of imaging devices 36 in accordance with the viewpoint information 74 acquired by the viewpoint information acquisition unit 28B. After the processing of step ST16 is executed, the image generation processing proceeds to step ST18 (see FIG. 9).

[0110] In step ST18, the virtual viewpoint image generation unit 28C acquires imaging condition information 64A from the plurality of captured images 64 acquired in step ST16, and acquires position information 39 related to the advertising billboard 6 shown in the captured image 64 from the NVM 30 by referring to the acquired imaging condition information 64A (see FIG. 9). After the processing of step ST18 is executed, the image generation processing proceeds to step ST20.

[0111] In step ST20, the virtual viewpoint image generation unit 28C determines viewpoint information 74 based on the position information 39 acquired in step ST18 so that the advertising billboard image 77 is included in the virtual viewpoint video 78 (see FIG. 9). After the processing of step ST20 is executed, the image generation processing proceeds to step ST22.

[0112] In step ST22, the virtual viewpoint image generation unit 28C generates the virtual viewpoint video 78 based on the viewpoint information 74 determined in step ST20 (see FIG. 11). After the processing of step ST22 is executed, the image generation processing proceeds to step ST24.

[0113] In step ST24, the virtual viewpoint image generation unit 28C transmits the virtual viewpoint video 78 generated in step ST22 to the user device 12 via the transmission / reception device 24 (see FIG. 12).

[0114] When the processing of step ST24 is executed and the virtual viewpoint video 78 is transmitted from the image processing device 10 to the user device 12, the user device 12 receives the virtual viewpoint video 78 and displays the received virtual viewpoint video 78 on the virtual viewpoint video screen 68 of the display 18 (see FIG. 13). The virtual viewpoint video 78 displayed on the virtual viewpoint video screen 68 is viewed by the user 14. After the processing of step ST24 is executed, the image generation processing proceeds to step ST26.

[0115] In step ST26, the virtual viewpoint image generation unit 28C determines whether or not a condition for terminating the image generation process (hereinafter referred to as the "termination condition") has been satisfied. A first example of the termination condition is that an instruction to terminate the image generation process has been accepted by the accepting device 50 (see FIG. 3). A second example of the termination condition is that communication between the image processing device 10 and one or more predetermined image capture devices 36 among the multiple image capture devices 36 has been interrupted. A third example of the termination condition is that a second predetermined time (e.g., 60 seconds) has elapsed without the judgment in step ST14 being affirmative.

[0116] In step ST26, if the termination condition is not satisfied, the determination is negative and the image generation process proceeds to step ST 10. In step ST26, if the termination condition is satisfied, the determination is positive and the image generation process ends.

[0117] As described above, the image processing device 10 according to this embodiment acquires a virtual viewpoint video 78 including an advertising billboard image 77 based on the position information 39 of the advertising billboard 6 captured in the captured image 64 and the viewpoint information 74. The virtual viewpoint video 78 acquired in this manner is then displayed on the display 18. Because the virtual viewpoint video 78 includes the advertising billboard image 77, the advertising billboard image 77 can be shown to the user 14 who is viewing the virtual viewpoint video 78.

[0118] Furthermore, in the image processing device 10 according to this embodiment, viewpoint information 74 is acquired by being received by viewpoint information acquisition unit 28B, and the received viewpoint information 74 is determined based on position information 39. Here, determining means, for example, that viewpoint information 74 is restricted. By determining viewpoint information 74 based on position information 39 in this manner, advertising billboard image 77 is included in virtual viewpoint video 78. Therefore, according to this configuration, advertising billboard image 77 can be easily included in virtual viewpoint video 78 compared to when only virtual viewpoint image 76 is generated regardless of the position of advertising billboard 6.

[0119] Furthermore, in the image processing device 10 according to this embodiment, the viewpoint path information 74A is determined based on the position information 39. Here, determining means, for example, restricting the viewpoint path information 74A. Restricting the viewpoint path information 74A means, for example, changing the viewpoint path information 74A related to viewpoint path P1 to viewpoint path information 74A related to viewpoint path P2 (see FIG. 9). By determining the viewpoint path information 74A based on the position information 39 in this way, the advertising billboard image 77 is included in the virtual viewpoint video 78. Therefore, according to this configuration, the advertising billboard image 77 can be more easily included in the virtual viewpoint video 78 than when only the virtual viewpoint image 76 is generated regardless of the position of the advertising billboard 6.

[0120] Furthermore, in the image processing device 10 according to this embodiment, the virtual viewpoint video 78 is displayed on the display 18 for a time period determined in accordance with the viewpoint information 74. Therefore, according to this configuration, the advertising billboard image 77 included in the virtual viewpoint video 78 can be shown to the user 14 for a time period determined in accordance with the viewpoint information 74.

[0121] [First Modification] In the above embodiment, an example has been described in which the viewpoint information 74 is limited to a range in which the position identified from the position information 39 by the virtual viewpoint image generation unit 28C is included in the area indicated by the virtual viewpoint image 76 (the area identified from the viewpoint information 74). However, the technology of the present disclosure is not limited to this. For example, the viewpoint information acquisition unit 28B may receive the viewpoint information 74 within a range in which the position identified from the position information 39 is included in the area indicated by the virtual viewpoint image 76 (the area identified from the viewpoint information 74), thereby acquiring the viewpoint information 74. Similarly, the point of gaze GP may be received within a range in which the position identified from the position information 39 is included in the area indicated by the virtual viewpoint image 76 (the area identified from the viewpoint information 74). In this case, as in the above embodiment, a virtual viewpoint video 78 is generated in accordance with the viewpoint information 74 and displayed on the display 18, so that the advertising billboard image 77 can be shown to the user 14, who is viewing the virtual viewpoint video 78.

[0122] [Second Modification] In the above embodiment, the first control is exemplified as a control in which the advertising billboard image 77 is included in the virtual viewpoint image 76 regardless of the size and position of the advertising billboard image 77 in the virtual viewpoint image 76, but the technology of the present disclosure is not limited to this. For example, as the first control, the virtual viewpoint image generation unit 28C may perform control in which the advertising billboard image 77 is included in the virtual viewpoint image 76 by determining the viewpoint information 74 based on the position information 39 to the extent that both the size and position of the advertising billboard image 77 in the virtual viewpoint image 76 satisfy the first predetermined condition. Here, the first predetermined condition is an example of a "second condition" according to the technology of the present disclosure. An example of the first predetermined condition is that the virtual viewpoint image 76 is divided into two in the vertical direction and two in the horizontal direction, resulting in an upper left region, an upper right region, a lower left region, and a lower right region, and 80% or more of the advertising billboard image 77 is located in a specific area (the upper right region in the example shown in FIG. 15 ) among these regions, and the advertising billboard image 77 has an occupancy rate of 15% or more in the virtual viewpoint image 76 (= size of the advertising billboard image 77 / overall size of the virtual viewpoint image 76). Note that the method of determining the divided regions and the occupancy rate exemplified here are merely examples. For example, if the area of ​​the virtual viewpoint image 76 is divided into a central area and a peripheral area, the first predetermined condition may be that the advertising billboard image 77 is located in the peripheral area, and the entire advertising billboard image 77 is sized so that it can fit into the peripheral area. Furthermore, the condition regarding the proportion of the billboard image 77 to be positioned in a specific area and / or the condition regarding the size of the billboard image 77 may be a fixed value, or may be a variable value that is changed according to instructions received by the receiving device 50 and / or various conditions (e.g., imaging conditions), etc.

[0123] Also, although an example has been given here in which both the size and position of the advertising billboard image 77 in the virtual viewpoint image 76 satisfy the first predetermined condition, this is merely one example, and the virtual viewpoint image generation unit 28C may perform, as a first control, control to include the advertising billboard image 77 in the virtual viewpoint image 76 by determining the viewpoint information 74 based on the position information 39 to the extent that either the size or the position of the advertising billboard image 77 in the virtual viewpoint image 76 satisfies the first predetermined condition.

[0124] According to this second modified example, compared to when the first control is performed regardless of the size and position of the advertising billboard image 77 in the virtual viewpoint image 78, it is possible to prevent the advertising billboard image 77 from being included in the virtual viewpoint image 78 at a size and / or position that is not intended by the party providing the advertising billboard image 77 to the user 14 (e.g., the advertiser).

[0125] [Third Modification] In the above embodiment, the first control is exemplified as a control for including the billboard image 77 in the virtual viewpoint image 76 regardless of the attributes of the billboard 6, but the technology of the present disclosure is not limited to this. For example, display priorities may be assigned to multiple billboards 6 (see FIG. 2), and the virtual viewpoint image generation unit 28C may perform the first control based on the display priorities when multiple billboards 6 with display priorities assigned appear in the captured image 64. Here, the display priorities refer to, for example, the priorities of display on the display 18. Hereinafter, for convenience of explanation, the display priorities will be simply referred to as "priorities."

[0126] 16 as an example, priority information 80 is stored in the NVM 30. The priority information 80 is information indicating the priorities of a plurality of advertising billboards 6. The virtual viewpoint image generation unit 28C acquires the priority information 80 from the NVM 30. Then, as a first control, the virtual viewpoint image generation unit 28C performs control to include an advertising billboard image 77 in a virtual viewpoint image 76 by referring to the priority information 80 acquired from the NVM 30 and determining viewpoint information 74 based on position information 39.

[0127] For example, if viewpoint path P1 is divided into a range in which a billboard image 77 showing the billboard 6 with the highest priority is generated and a range in which a billboard image 77 showing the billboard 6 with the highest priority is not generated but a billboard image 77 showing the billboard 6 with the tenth priority is generated, virtual viewpoint image generation unit 28C generates viewpoint path information 74A indicating viewpoint path P3 that limits viewpoint path P1 to only the range in which a billboard image 77 showing the billboard 6 with the highest priority is generated. In this case, virtual viewpoint image generation unit 28C generates a virtual viewpoint video 78 including a billboard image 77 showing the billboard 6 with the highest priority. The virtual viewpoint video 78 generated in this manner is displayed on display 18. Furthermore, for example, if a portion of the billboard image 77 showing the billboard 6 with the first priority and a portion of the billboard image 77 showing the billboard 6 with the tenth priority are generated at a certain viewpoint position on the viewpoint path P1, the virtual viewpoint image generation unit 28C may change the viewpoint position to a viewpoint position where all of the billboard images 77 showing the billboard 6 with the first priority are generated, but where no billboard images 77 showing the billboard 6 with the tenth priority are generated.

[0128] According to the third modified example, when a plurality of billboards 6 appear in the captured image 64, the user 14 can be shown the billboard images 77 showing the billboards 6 determined in order of priority.

[0129] Furthermore, in this third modified example, the priority order is determined based on the attributes of the billboard 6. One example of the attributes of the billboard 6 is the charge for the billboard 6. In other words, the higher the charge for the billboard 6, the higher the priority order. In this case, the billboard image 77 can be shown to the user 14 according to the priority order determined based on the attributes of the billboard 6.

[0130] The attributes of the billboard 6 are not limited to charges, but may be, for example, the size of the billboard 6, the type of advertiser of the billboard 6, information depicted on the billboard 6, and / or the background color of the billboard 6.

[0131] [Fourth Modification] In the third modified example, the priority order is determined based on the attributes of the billboard 6, but the technology of the present disclosure is not limited to this. For example, the priority order may be determined based on the attributes of the user 14 who set the viewpoint information 74.

[0132] 17, for example, user attribute information 74G indicating the attributes of the user 14 who set the viewpoint information 74 (for example, the user 14 who specified a viewpoint in the viewpoint setting mode and a gaze point in the gaze point setting mode) is included in the viewpoint information 74. The attributes of the user 14 refer to, for example, the gender, age, hometown, team supported, occupation, clothing, race, belongings, and / or the name of a favorite player of the user 14.

[0133] The virtual viewpoint image generation unit 28C extracts user attribute information 74G from the viewpoint information 74. Then, the virtual viewpoint image generation unit 28C determines the priority order based on the user attribute information 74G extracted from the viewpoint information 74. For example, if the attribute of the user 14 is female, by giving a higher priority to billboards 6 that suit women's preferences than to other billboards 6, the virtual viewpoint video 78 is more likely to include billboard images 77 showing billboards 6 that suit women's preferences than billboard images 77 showing billboards 6 other than the billboards 6 that suit women's preferences.

[0134] According to the fourth modified example, it is possible to make it easier for the user 14 to see the advertising billboard image 77 that suits the preferences of the user 14, compared to when the priority is determined regardless of the attributes of the user 14 who set the viewpoint information 74.

[0135] [Fifth Modification] In the fourth modified example, an example in which the priority is determined based on the attributes of the user 14 has been described, but the technology of the present disclosure is not limited to this. For example, the priority may be determined based on the status of the subject imaged by the multiple imaging devices 36.

[0136] In this case, as an example, as shown in Fig. 18, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64. The virtual viewpoint image generation unit 28C identifies the situation of the imaging target by recognizing the subjects appearing in the plurality of captured images 64. Examples of subject recognition methods include a subject recognition method using AI, a subject recognition method using template matching, and the like. The virtual viewpoint image generation unit 28C determines a priority order based on the identified situation of the imaging target.

[0137] Here, the situation of the imaging subject refers to, for example, the degree of crowding of multiple moving objects (e.g., multiple people). The degree of crowding is expressed, for example, by population density. For example, as priorities for multiple advertising billboards 6, a priority when the degree of crowding is higher than a preset value and a priority when the degree of crowding is equal to or lower than the preset value are predefined, and the virtual viewpoint image generation unit 28C determines whether to use the priority when the degree of crowding is higher than the preset value or the priority when the degree of crowding is equal to or lower than the preset value according to the identified degree of crowding. The default value may be a fixed value or a variable value that is changed according to an instruction accepted by the accepting device 50 and / or various conditions (e.g., imaging conditions), etc.

[0138] According to the fifth modification, the advertising signboard images 77 can be shown to the user 14 in accordance with the priority order determined based on the status of the subjects imaged by the multiple imaging devices 36.

[0139] Note that although the degree of crowding is shown here as an example of the condition of the subject to be imaged, this is merely an example, and the condition of the subject to be imaged may also be, for example, the speed of the subject's movement, the size of the subject, the position of the subject, the brightness of the subject, and / or the type of subject, etc.

[0140] [Sixth Modification] In the above embodiment, the first control is exemplified as a control to include the advertising billboard image 77 in the virtual viewpoint image 76, but if an image showing only the back side of the advertising billboard 6 (for example, the side that does not depict any information that the advertiser wants to communicate) is included in the virtual viewpoint image 76 as the advertising billboard image 77, the advertising effect intended by the advertiser cannot be expected.

[0141] 19, the virtual viewpoint image generation unit 28C changes the display mode of the billboard image 77 based on the viewpoint information 74 and the position information 39. In this case, for example, the virtual viewpoint image generation unit 28C refers to the viewpoint information 74 and the position information 39 and determines whether or not an image showing only the back side of the billboard 6 is included in the virtual viewpoint image 76 as the billboard image 77. If the virtual viewpoint image generation unit 28C determines that an image showing only the back side of the billboard 6 is included in the virtual viewpoint image 76 as the billboard image 77, the virtual viewpoint image generation unit 28C performs control to include in the virtual viewpoint image 76 an image showing the front side of the billboard 6 (for example, the side on which the information that the advertiser wants to transmit is depicted).

[0142] In the sixth modified example, the user 14 can be shown the advertising billboard image 77 whose display mode has been changed based on the viewpoint information 74 and the position information 39.

[0143] Here, an example has been given in which virtual viewpoint image generation unit 28C changes the orientation of advertising billboard image 77, but the technology of the present disclosure is not limited to this. Virtual viewpoint image generation unit 28C may refer to viewpoint information 74 and position information 39, and if the size of advertising billboard image 77 included in virtual viewpoint image 76 is smaller than a reference size, change the size of advertising billboard image 77 to a size equal to or larger than the reference size or emphasize the outline of advertising billboard image 77, thereby changing the display mode of advertising billboard image 77.

[0144] [Seventh Modification] In the above embodiment, the first control is exemplified, but this is merely an example, and the second control may be performed instead of or together with the first control. Here, the second control refers to a control in which the advertising billboard image 77 is included in the virtual viewpoint image 76 by moving the advertising billboard image 77 based on the position information 39.

[0145] As an example, as shown in Figure 20, when the virtual viewpoint image generation unit 28C acquires multiple captured images 64 according to viewpoint information 74 (in the example shown in Figure 20, information including viewpoint path information 74A indicating viewpoint path P1) and generates a virtual viewpoint video 78 based on the acquired multiple captured images 64, the virtual viewpoint image 76 of at least the final frame included in the virtual viewpoint video 78 may not include an advertising billboard image 77.

[0146] Therefore, in such a case, the virtual viewpoint image generation unit 28C performs the second control. In this case, as shown in FIG. 21 as an example, first, the virtual viewpoint image generation unit 28C acquires, from the NVM 30, position information 39 related to the advertising billboard 6 indicated by the advertising billboard image 77 included in the virtual viewpoint image 76 other than the virtual viewpoint image 76 that does not include the advertising billboard image 77 in the virtual viewpoint video 78. Next, the virtual viewpoint image generation unit 28C acquires viewpoint information 74 used to generate the virtual viewpoint image 76 that does not include the advertising billboard image 77 (for example, the virtual viewpoint image 76 of the final frame). Then, the virtual viewpoint image generation unit 28C refers to the position information 39 and the viewpoint information 74 and moves the advertising billboard image 77 in the virtual viewpoint image 76 of the other frame into the virtual viewpoint image 76 that does not include the advertising billboard image 77. For example, virtual viewpoint image generation unit 28C moves billboard image 77, which indicates billboard 6 located closest to the angle of view specified from angle of view information 74F (see FIG. 8 ) included in viewpoint information 74 used to generate virtual viewpoint image 76 that does not include billboard image 77 (for example, virtual viewpoint image 76 of the final frame), into virtual viewpoint image 76 that does not include billboard image 77. This makes it easier to include billboard image 77 in virtual viewpoint image 78 compared to when only virtual viewpoint video 78 is simply generated regardless of the position of billboard 6.

[0147] It is possible to apply the above-described second modification to the seventh modification. That is, the virtual viewpoint image generation unit 28C may perform the second control based on at least one of the size and the position of the advertising billboard image 77 in the virtual viewpoint image 76 generated based on the viewpoint information 74. In this case, compared to the case where the second control is performed regardless of either the size or the position of the advertising billboard image 77 in the virtual viewpoint video 78, it is possible to prevent the advertising billboard image 77 from being included in the virtual viewpoint video 78 at a size and / or position that is not intended by the party providing the user 14 (for example, an advertiser).

[0148] The above-described third modification can be applied to the seventh modification. That is, priorities are assigned to multiple advertising billboards 6 (see FIG. 2), and the virtual viewpoint image generating unit 28C may perform the second control based on the priorities when multiple advertising billboards 6 with assigned priorities appear in the captured image 64. In this case, the same effect as in the third modification can be obtained.

[0149] The priority order may be determined based on the attributes of the billboard 6. In the seventh modified example, the priority order may be determined by the methods described in the fourth and fifth modified examples. In the seventh modified example, the display mode of the billboard image 77 may be changed by the method described in the sixth modified example.

[0150] [Eighth Modification] In the above embodiment, the first control is exemplified, but this is merely an example, and the third control may be performed instead of or together with the first control. Furthermore, the third control may be performed instead of or together with the second control described in the seventh modified example. Here, the third control refers to a control in which the viewpoint information 74 is changed based on the position information 39 to include an advertising billboard image 77 in the virtual viewpoint image 76.

[0151] 22 as an example, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from a plurality of imaging devices 36 in accordance with viewpoint information 74 received by the viewpoint information acquisition unit 28B, similarly to the above embodiment. That is, the virtual viewpoint image generation unit 28C acquires a plurality of captured images 64 from the plurality of imaging devices 36 to be used for generating a virtual viewpoint image 76 (see FIG. 11 ) that shows the state of the subject when the subject is observed in accordance with the viewpoint information 74. The virtual viewpoint image generation unit 28C refers to imaging condition information 64A included in each captured image 64, and acquires, from the NVM 30, position information 39 regarding the advertising billboard 6 that is entirely captured in the captured image 64.

[0152] As a third control, the virtual viewpoint image generation unit 28C determines the viewpoint information 74 received by the viewpoint information acquisition unit 28B based on the position information 39, thereby controlling the inclusion of an advertising billboard image 77 in the virtual viewpoint image 76. In the eighth modified example, an example of determining the viewpoint information 74 is changing the viewpoint path information 74A. That is, the virtual viewpoint image generation unit 28C changes the viewpoint path information 74A so that the advertising billboard 6 appears in the virtual viewpoint image 76. In this case, for example, as shown in FIG. 22, the viewpoint path P1 changes to the viewpoint path P4. Also, the start point position information and / or the end point position information in the viewpoint path information 74A may be changed. Note that, although the change in the viewpoint path information 74A is illustrated here, this is merely an example, and the line of sight direction information 74B and / or the angle of view information 74F may be changed so that the advertising billboard 6 appears in the virtual viewpoint image 76.

[0153] According to the eighth modified example, the billboard image 77 can be included in the virtual viewpoint video 78 more easily than when only the virtual viewpoint video 78 is generated regardless of the position of the billboard 6.

[0154] The above-described third modified example can be applied to the eighth modified example. That is, priorities are assigned to multiple advertising billboards 6 (see FIG. 2), and the virtual viewpoint image generating unit 28C may perform the third control based on the priorities when multiple advertising billboards 6 with assigned priorities appear in the captured image 64. In this case, the same effect as the third modified example can be obtained.

[0155] When the above-described third modification is applied to the eighth modification, the priority may be determined based on the attributes of the billboard 6. In addition, in the eighth modification, the priority may be determined by the methods described in the fourth and fifth modifications. In addition, in the eighth modification, the display mode of the billboard image 77 may be changed by the method described in the sixth modification.

[0156] The above-described second modification can also be applied to the eighth modification. That is, the virtual viewpoint image generation unit 28C may perform the third control based on at least one of the size and the position of the advertising billboard image 77 in the virtual viewpoint image 76 generated based on the viewpoint information 74. In this case, compared to the case where the second control is performed regardless of either the size or the position of the advertising billboard image 77 in the virtual viewpoint video 78, it is possible to prevent the advertising billboard image 77 from being included in the virtual viewpoint video 78 at a size and / or position that is not intended by the party providing the user 14 (for example, an advertiser).

[0157] [Ninth Variation] Although the eighth modified example illustrates an example of changing the viewpoint path information 74A, the technology of the present disclosure is not limited to this. For example, the line of sight direction information 74B may be changed based on the position information 39 at the position of the start point P1s and / or the position of the end point P1e of the viewpoint path P1 indicated by the viewpoint path information 74A. In this case, for example, as shown in FIG. 23, the virtual viewpoint image generation unit 28C changes the line of sight at the position of the end point P1e by referring to the position information 39 so that an advertising signboard image 77 is included in the virtual viewpoint image 76 at the position of the end point P1e. ​​In the example shown in FIG. 23, the line of sight direction is changed at the position of the end point P1e, but the line of sight direction may also be changed at the position of the start point P1s.

[0158] According to the present ninth variant, it is easier to include the advertising billboard image 77 in the virtual viewpoint video 78 than when the line of sight is fixed in one direction at the position of the starting point P1s of the viewpoint path P1 and the position of the ending point P1e of the viewpoint path P1.

[0159] The above-described third modified example can be applied to the ninth modified example. That is, priorities are assigned to multiple advertising billboards 6 (see FIG. 2), and the virtual viewpoint image generating unit 28C may perform the third control based on the priorities when multiple advertising billboards 6 with assigned priorities appear in the captured image 64. In this case, the same effect as the third modified example can be obtained.

[0160] When the above-described third modification is applied to the ninth modification, the priority may be determined based on the attributes of the advertising billboard 6. Furthermore, in the ninth modification, the priority may be determined by the methods described in the fourth and fifth modifications. Furthermore, in the ninth modification, the display mode of the advertising billboard image 77 may be changed by the method described in the sixth modification.

[0161] Note that, in the ninth modified example, an example in which the line of sight direction is changed at the position of the start point P1s and / or the position of the end point P1e of the viewpoint path P1 has been described. However, the technology of the present disclosure is not limited to this, and the line of sight direction may be changed at a position along the viewpoint path P1. For example, if the required time indicated by the required time information 74C (see FIG. 8) is equal to or greater than a first threshold (e.g., 30 seconds), the virtual viewpoint image generation unit 28C may change the line of sight direction by referring to the position information 39 so that the advertising billboard image 77 is included in the virtual viewpoint image 76 before the required time has elapsed. Furthermore, for example, if the elapsed time indicated by the elapsed time information 74D (see FIG. 8) is equal to or greater than a second threshold (e.g., 10 seconds), the virtual viewpoint image generation unit 28C may change the line of sight direction by referring to the position information 39 so that the advertising billboard image 77 is included in the virtual viewpoint image 76 at the position of the viewpoint corresponding to the elapsed time information 74D indicating an elapsed time equal to or greater than the second threshold. Furthermore, for example, if the movement speed identified from the movement speed information 74E (see FIG. 8) is equal to or greater than a third threshold (e.g., 1 meter / second), the virtual viewpoint image generation unit 28C may change the line of sight direction by referring to the position information 39 so that the advertising billboard image 77 is included in the virtual viewpoint image 76 at a viewpoint position corresponding to the movement speed equal to or greater than the third threshold. Furthermore, for example, if the change amount per unit time of the angle of view identified from the angle of view information 74F (see FIG. 8) is equal to or greater than a fourth threshold (e.g., 30 degrees), the virtual viewpoint image generation unit 28C may change the line of sight direction by referring to the position information 39 so that the advertising billboard image 77 is included in the virtual viewpoint image 76 at a viewpoint position where the change amount per unit time is equal to or greater than the fourth threshold.

[0162] Furthermore, not only is the line of sight changed, but the size of the billboard image 77 in the virtual viewpoint image 76 may also be changed at the position of the viewpoint where the line of sight is changed. The angle of view may also be changed. In this way, by displaying the billboard image 77 in an emphasized manner when the line of sight is changed, the advertising effect of the billboard image 77 on the user 14 can be increased compared to when the billboard image 77 is not displayed in an emphasized manner.

[0163] In the above example, when the virtual viewpoint image 76 does not include the advertising billboard image 77, the viewpoint information 74 is restricted, the advertising billboard image 77 is moved, or the viewpoint information 74 is changed. However, the technology of the present disclosure is not limited to this. For example, when the area identified based on the viewpoint information 74 received by the viewpoint information acquisition unit 28B does not include the position identified from the position information 39, i.e., the position of the advertising billboard 6, the virtual viewpoint image generation unit 28C may change the position information 39. In this case, even when the area identified based on the viewpoint information 74 does not include the position identified from the position information 39, the advertising billboard image 77 can be shown to the user 14. While the example of changing the position information 39 has been given here, the present disclosure is not limited to this. Instead of or in addition to changing the position information 39, the position of the advertising billboard image 77 may be changed.

[0164] Also, for example, when the viewpoint information 74 and position information 39 received by the viewpoint information acquisition unit 28B satisfy the first condition, the virtual viewpoint image generation unit 28C may change the position of the position information 39 and / or the advertising billboard image 77.

[0165] Here, a first example of the first condition is a condition that only a portion (e.g., 10% of the advertising billboard 6) of the billboard 6 identified from the position information 39 is included in an area identified based on the viewpoint information 74 received by the viewpoint information acquisition unit 28B. Furthermore, a second example of the first condition is a condition that the advertising billboard 6 identified from the position information 39 is included in a specific area (e.g., the center) of the area identified based on the viewpoint information 74 received by the viewpoint information acquisition unit 28B.

[0166] When the condition shown as a first example of the first condition is satisfied, for example, the virtual viewpoint image generation unit 28C changes the position of the position information 39 and / or the position of the advertising billboard image 77 so that the entire billboard 6 is included in the area specified based on the viewpoint information 74. Furthermore, when the condition shown as a second example of the first condition is satisfied, for example, the virtual viewpoint image generation unit 28C changes the position of the position information 39 and / or the advertising billboard image 77 so that the entire billboard 6 is included in the edge of the area specified based on the viewpoint information 74.

[0167] In this way, if the viewpoint information 74 and position information 39 received by the viewpoint information acquisition unit 28B satisfy the first condition, the virtual viewpoint image generation unit 28C changes the position of the position information 39 and / or the advertising billboard image 77. In this configuration, the advertising billboard image 77 can be shown to the user 14 in accordance with the relationship between the viewpoint information 74 and the position information 39.

[0168] Furthermore, in the above example, the virtual viewpoint image generation unit 28C generates a virtual viewpoint image 76, which is an image showing the appearance of the subject when observed from a viewpoint specified by the viewpoint information 74, based on the multiple captured images 64 and the viewpoint information 74. However, the technology of the present disclosure is not limited to this, and the virtual viewpoint image generation unit 28C may cause an external device (e.g., a server) communicatively connected to the image processing device 10 to generate the virtual viewpoint image 76, and may acquire the virtual viewpoint image 76 from the external device.

[0169] Furthermore, in the above example, an advertising billboard 6 is illustrated, but the technology of the present disclosure is not limited to this, and may be, for example, a soccer player's clothing (clothing with an advertisement), a balloon with an advertisement, a mascot with an advertisement, a flag with an advertisement, and / or an advertising screen projected by a projector, etc. It may also be a specific object without an advertisement (for example, a three-dimensional object protected as a trademark).

[0170] Furthermore, in the above example, an example was given in which the virtual viewpoint image 76 includes the entire advertising billboard image 77, but the technology of the present disclosure is not limited to this, and it is sufficient if the virtual viewpoint image 76 includes a portion (e.g., more than half) of the advertising billboard image 77.

[0171] Furthermore, in the above example, an example was given in which the image generation processing is performed by the computer 22 of the image processing device 10, but the technology of the present disclosure is not limited to this, and the image generation processing may be performed by the computer 40 of the user device 12, or distributed processing may be performed by the computer 22 of the image processing device 10 and the computer 40 of the user device 12.

[0172] Furthermore, while the computer 22 is illustrated in the above example, the technology of the present disclosure is not limited to this. For example, a device including an ASIC, FPGA, and / or PLD may be applied instead of the computer 22. Furthermore, a combination of hardware and software configurations may be used instead of the computer 22. The same applies to the computer 40 of the user device 12.

[0173] Furthermore, in the above example, the image generation processing program 38 is stored in the NVM 30, but the technology of the present disclosure is not limited to this, and as an example, the image generation processing program 38 may be stored in any portable storage medium 300 such as an SSD or USB memory, which is a non-transitory storage medium, as shown in Fig. 24. In this case, the image generation processing program 38 stored in the storage medium 300 is installed in the computer 22, and the CPU 28 executes the image generation processing in accordance with the image generation processing program 38.

[0174] Alternatively, the image generation processing program 38 may be stored in a memory of another computer or server device connected to the computer 22 via a communication network (not shown), and the image generation processing program 38 may be downloaded to the image processing device 10 in response to a request from the image processing device 10. In this case, the image generation processing is executed by the CPU 28 of the computer 22 in accordance with the downloaded image generation processing program 38.

[0175] Furthermore, in the above example, CPU28 is exemplified, but instead of CPU28 or together with CPU28, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0176] The hardware resources that execute the image generation process can be various processors, as listed below. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource that executes the image generation process according to software, i.e., a program, as described above. Other processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically to execute dedicated processes. Each processor has built-in or connected memory, and executes the image generation process using the memory.

[0177] The hardware resource that executes the image generation process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the image generation process may be a single processor.

[0178] Examples of systems using a single processor include: first, a system in which one processor is configured using a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes image generation processing; second, a system in which a processor is used to implement the functions of the entire system, including multiple hardware resources that execute image generation processing, on a single IC chip, as typified by SoCs; and image generation processing is thus implemented using one or more of the various processors described above as hardware resources.

[0179] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0180] Furthermore, the image generation process described above is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0181] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0182] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0183] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; a memory connected to or embedded in the processor; The processor: acquiring a first virtual viewpoint image generated based on a plurality of captured images; Obtain viewpoint information, acquiring position information of an object captured in the captured image; A second virtual viewpoint image including an object image showing the object is obtained based on the viewpoint information and the position information. Image processing device.

2. The processor: acquiring the viewpoint information by receiving the viewpoint information; The received viewpoint information is determined based on the position information, thereby performing a first control of including the object image in the second virtual viewpoint image. The image processing device according to claim 1 .

3. The viewpoint information includes the received first viewpoint path. The image processing device according to claim 2 .

4. the viewpoint information is information that identifies an area shown by the second virtual viewpoint image, The processor acquires the viewpoint information by accepting the viewpoint information within a range in which a position specified by the position information is included in the area. The image processing device according to claim 2 .

5. The processor: When the position identified from the position information is not included in the area identified based on the viewpoint information, Changing at least one of the position information and the position of the object image The image processing device according to claim 1 .

6. The processor: When the viewpoint information and the position information satisfy a first condition, Changing at least one of the position information and the position of the object image The image processing device according to claim 1 .

7. The processor performs second control to include the object image in the second virtual viewpoint image by moving the object image based on the position information. The image processing device according to any one of claims 1 to 6.

8. The processor performs third control to include the object image in the second virtual viewpoint image by changing the viewpoint information based on the position information. The image processing device according to any one of claims 1 to 7.

9. The viewpoint information includes at least one of start point position information capable of specifying the position of the start point of the second viewpoint path, end point position information capable of specifying the position of the end point of the second viewpoint path, first line of sight direction information capable of specifying a first line of sight direction, and view angle information capable of specifying a view angle. The image processing device according to claim 8 .

10. the viewpoint information includes second line-of-sight direction information that can identify a second line-of-sight direction, The third control includes control of including the object image in the second virtual viewpoint image by changing the second line of sight information based on position information of at least one of a position of a start point of a third viewpoint path and a position of an end point of the third viewpoint path as the viewpoint information.

10. The image processing device according to claim 8 or claim 9.

11. the second virtual viewpoint image includes a first object image showing an object, The processor performs the first control within a range in which at least one of a size and a position of the first object image in the second virtual viewpoint image satisfies a second condition. The image processing device according to any one of claims 2 to 4.

12. The processor performs the second control based on at least one of a size and a position of a second object image showing an object in a third virtual viewpoint image generated based on the viewpoint information. The image processing device according to claim 7 .

13. The processor performs the third control based on at least one of a size and a position of a third object image showing an object in a third virtual viewpoint image generated based on the viewpoint information. The image processing device according to any one of claims 8 to 10.

14. The objects are given display priorities; When the plurality of objects to which the priorities have been assigned appear in the captured image, the processor performs the first control based on the priorities.

12. The image processing device according to claim 2, claim 3, claim 4, or claim 11.

15. The objects are given display priorities; When the plurality of objects to which the priorities have been assigned appear in the captured image, the processor performs the second control based on the priorities.

13. The image processing device according to claim 7 or 12.

16. The objects are given display priorities; When the plurality of objects to which the priorities have been assigned appear in the captured image, the processor performs the third control based on the priorities.

14. The image processing device according to claim 8, claim 9, claim 10, or claim 13.

17. The priority is determined based on the attributes of the object. The image processing device according to any one of claims 14 to 16.

18. The processor determines the priority based on the attributes of the user who set the viewpoint information.

18. The image processing device according to claim 14.

19. The processor determines the priority based on the status of the object to be imaged by the plurality of image capturing devices.

19. The image processing device according to any one of claims 14 to 18.

20. The processor changes the display mode of the object image based on the viewpoint information and the position information.

20. The image processing device according to claim 1.

21. The processor outputs data for displaying the second virtual viewpoint image on a display for a time period determined in accordance with the viewpoint information.

21. The image processing device according to claim 1.

22. acquiring a first virtual viewpoint image generated based on a plurality of captured images; Obtaining viewpoint information, Acquiring position information of an object captured in the captured image; and acquiring a second virtual viewpoint image including an object image showing the object based on the viewpoint information and the position information; An image processing method comprising:

23. On the computer, acquiring a first virtual viewpoint image generated based on a plurality of captured images; Obtaining viewpoint information, Acquiring position information of an object captured in the captured image; and a program for executing a process including acquiring a second virtual viewpoint image including an object image showing the object based on the viewpoint information and the position information;

Images