Information processing device, operation method of information processing device, and program

Abstract

To reduce the difference in the field of view between an image captured by a reference imaging device and a virtual viewpoint image.SOLUTION: An information processing device includes a processor and a memory connected to or built into the processor. The processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and generates a virtual viewpoint image on the basis of the reference imaging device information on condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging area by the plurality of imaging devices is given.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing apparatus, a method for operating the information processing apparatus, and a program.

Background Art

[0002] International Publication WO2018 / 021067 discloses an image processing apparatus including a virtual viewpoint image generation unit that generates a virtual viewpoint image captured at a virtual viewpoint composed of a virtual viewpoint position and a viewing direction from a plurality of reference images captured at a plurality of viewpoint positions and viewing directions.

[0003] Japanese Unexamined Patent Application Publication No. 2019-036790 discloses an image generation apparatus including: an image acquisition unit that acquires a group of images of a subject captured from a plurality of different positions; a viewpoint acquisition unit that acquires position information of a virtual viewpoint; a setting unit that sets the resolution of subject shape data according to the distance between a reference point serving as a reference for resolution setting and the virtual viewpoint; and a generation unit that generates a virtual viewpoint image of the subject from the virtual viewpoint using the subject shape data according to the resolution and the group of images of the subject.

Summary of the Invention

[0004] One embodiment of the technology of the present disclosure provides an information processing apparatus, a method for operating the information processing apparatus, and a program that can reduce the difference in the field of view between an image captured by a reference imaging device and a virtual viewpoint image.

Means for Solving the Problems

[0005] A first aspect of the technology of the present disclosure is an information processing apparatus including a processor and a memory connected to or built in the processor, wherein the processor acquires reference imaging device information corresponding to the position, imaging direction, and angle of view of a reference imaging device, and generates a virtual viewpoint image based on the reference imaging device information on the condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region by a plurality of imaging devices is given.

[0006] A second aspect of the technology according to the present disclosure is an information processing apparatus according to the first aspect, in which when reference imaging device information is continuously transmitted by a transmission device, a processor generates a virtual viewpoint image based on the reference imaging device information transmitted from the transmission device on the condition that an instruction is given.

[0007] A third aspect of the technology according to the present disclosure is an information processing apparatus according to the first aspect or the second aspect, in which a processor updates the reference imaging device information each time the reference imaging device information is acquired.

[0008] A fourth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to third aspects, in which a processor acquires image quality information indicating the image quality of a first reference image obtained by imaging an imaging area with a reference imaging device, and determines the image quality of a virtual viewpoint image based on the image quality information on the condition that an instruction is given.

[0009] A fifth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to fourth aspects, in which a processor outputs a superimposed image in which a virtual viewpoint image is superimposed on a second reference image obtained by imaging an imaging area with a reference imaging device on the condition that an instruction is given.

[0010] A sixth aspect of the technology according to the present disclosure is an information processing apparatus according to the fifth aspect, in which a processor gradually changes the ratio between the second reference image and the virtual viewpoint image in the superimposed image. the virtual viewpoint image in the superimposed image.

[0011] A seventh aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to sixth aspects, in which a processor outputs a virtual viewpoint image to a display and receives a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, and an angle of view in the output virtual viewpoint image.

[0012] The eighth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to seventh aspects, wherein the reference imaging device is an imaging device capable of changing at least one of a position, an imaging direction, and an angle of view.

[0013] The ninth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to eighth aspects, wherein the processor acquires reference imaging device information based on a third reference image obtained by imaging an imaging region with the reference imaging device.

[0014] The tenth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to ninth aspects, wherein the reference imaging device information is the position, the imaging direction, and the angle of view of the reference imaging device.

[0015] The eleventh aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to tenth aspects, wherein the reference imaging device is one of a plurality of imaging devices.

[0016] The twelfth aspect of the technology according to the present disclosure is an information processing apparatus according to the eleventh aspect, wherein the reference imaging device is switchable among a plurality of imaging devices.

[0017] The thirteenth aspect of the technology according to the present disclosure is an information processing apparatus according to any one of the first to twelfth aspects, wherein the processor outputs a fourth reference image obtained by imaging an imaging region with the reference imaging device, acquires reference imaging device information, and generates a virtual viewpoint image corresponding to the fourth reference image based on the reference imaging device information on the condition that an instruction is given.

[0018] A fourteenth aspect of the technology of the present disclosure is a method of operating an information processing apparatus including a processor and a memory connected to or incorporated in the processor, the method including: obtaining reference imaging apparatus information corresponding to the position, imaging direction, and field angle of a reference imaging apparatus; and generating a virtual viewpoint image based on the reference imaging apparatus information on the condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging apparatuses is given.

[0019] A fifteenth aspect of the technology of the present disclosure is a program for causing a computer applied to an information processing apparatus including a processor and a memory connected to or incorporated in the processor to execute a process including: obtaining reference imaging apparatus information corresponding to the position, imaging direction, and field angle of a reference imaging apparatus; and generating a virtual viewpoint image based on the reference imaging apparatus information on the condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging apparatuses is given.

Brief Description of Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0021] An example of an embodiment according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] CPU refers to the abbreviation of "Central Processing Unit". RAM refers to the abbreviation of "Random Access Memory". DRAM refers to the abbreviation of "Dynamic Random Access Memory". SRAM refers to the abbreviation of "Static Random Access Memory". ROM refers to the abbreviation of "Read Only Memory". SSD refers to the abbreviation of "Solid State Drive". HDD refers to the abbreviation of "Hard Disk Drive". EEPROM refers to the abbreviation of "Electrically Erasable and Programmable Read Only Memory". I / F refers to the abbreviation of "Interface". IC refers to the abbreviation of "Integrated Circuit". ASIC refers to the abbreviation of "Application Specific Integrated Circuit". PLD refers to the abbreviation of "Programmable Logic Device". FPGA refers to the abbreviation of "Field-Programmable Gate Array" and SoC refers to the abbreviation of "System-on-a-chip". CMOS refers to "Complement" refers to the abbreviation of "ary Metal Oxide Semiconductor". CCD refers to the abbreviation of "Charge Coupled Device". EL refers to the abbreviation of "Electro-Luminescence". GPU refers to the abbreviation of "Graphics Processing Unit". LAN refers to the abbreviation of "Local Area Network". 3D refers to the abbreviation of "3 Dimension". "HMD" refers to the abbreviation of "Head Mounted Display". USB refers to the abbreviation of "Universal Serial Bus". fps refers to , the abbreviation of "frame per second". LTE refers to the abbreviation of "Long Term Evolution" refers to. 5G refers to the abbreviation of "5th generation (wireless technology for digital cellular networks)". TDM refers to the abbreviation of "Time-Division Multiplexing".

[0024] In the description of this specification, the meaning of "identical" includes not only the meaning of complete identity, but also the meaning of approximate identity including errors acceptable in design and manufacture. In the description of this specification, the meaning of "the same" includes not only the meaning of complete sameness, but also the meaning of approximate sameness including errors acceptable in design and manufacture. In the description of this specification, the meaning of "linear" includes not only the meaning of a complete straight line, but also the meaning of an approximate straight line including errors acceptable in design and manufacture. Also, in the description of this specification, the meaning of "curvilinear" includes not only the meaning of a complete curve, but also the meaning of an approximate curve including errors acceptable in design and manufacture.

[0025] As an example, as shown in FIG. 1, the information processing system 10 includes an information processing apparatus 12, a smart device 14, a plurality of imaging devices 16, an imaging device 18, a wireless communication base station (hereinafter simply referred to as "base station") 20, and a receiver 34. Here, the "smart device 14" refers to a portable multifunctional terminal such as a smartphone tablet terminal, a smartwatch (watch-type multifunctional terminal), and an HMD-type multifunctional terminal. Here, the information processing apparatus 12 is an example of the "information processing apparatus" according to the technology of the present disclosure. The plurality of imaging devices 16 and the imaging device 18 are examples of the "plurality of imaging devices" according to the technology of the present disclosure. Here, the receiver 34 is illustrated, but the technology of the present disclosure is not limited thereto, and it may be an electronic device with a display (for example, a smart device). Further, the base station 20 is not limited to one location and may be plural. Furthermore, the communication standards used in the base station include wireless communication standards including the LTE standard and the like, wireless communication standards including the WiFi (802.11) standard and / or the Bluetooth (registered trademark) standard, and wired communication standards including the TDM standard and / or the Ethernet (registered trademark) standard.

[0026] The imaging devices 16 and 18 are imaging devices having a CMOS image sensor, and are equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors such as a CCD image sensor may be employed instead of the CMOS image sensor. Hereinafter, for convenience of explanation, when it is not necessary to distinguish between the imaging device 18 and the plurality of imaging devices 16, they are referred to as the "plurality of imaging devices" without reference numerals.

[0027] The plurality of imaging devices 16 are installed inside the soccer stadium 22. Each of the plurality of imaging devices 16 is arranged to surround the soccer field 24, and images an area including the soccer field 24 as an imaging area. Here, the area including the soccer field 24 is an example of the "imaging area" according to the technology of the present disclosure. Here, a configuration example is given in which each of the plurality of imaging devices 16 is arranged to surround the soccer field 24, but the technology of the present disclosure is not limited to this, and the arrangement of the plurality of imaging devices 16 is determined according to the virtual viewpoint video required to be generated by the viewer 28 or the like. The plurality of imaging devices 16 may be arranged to surround the entire soccer field 24, or the plurality of imaging devices 16 may be arranged to surround a specific part. The imaging device 18 is installed on an unmanned aerial vehicle (for example, a multi-rotor type unmanned aerial vehicle), and images the area including the soccer field 24 from above in a state of overlooking from above. The imaging area in a state of overlooking the area including the soccer field 24 from above refers to the imaging surface of the imaging device 18 with respect to the soccer field 24.

[0028] The information processing device 12 is installed in the control room 32. The plurality of imaging devices 16 and the information processing device 12 are connected via a LAN cable 30. The information processing device 12 controls the plurality of imaging devices 16 and acquires images obtained by being imaged by each of the plurality of imaging devices 16. Here, a connection using a wired communication method via the LAN cable 30 is exemplified, but it is not limited to this, and a connection using a wireless communication method may also be used.

[0029] In the soccer stadium 22, spectator seats 26 are provided so as to surround the soccer field 24, and viewers 28 are seated in the spectator seats 26. The viewer 28 holds a smart device 14, and the smart device 14 is used by the viewer 28. Here, a configuration example in which the viewer 28 exists inside the soccer stadium 22 is described, but the technology of the present disclosure is not limited to this, and the viewer 28 may exist outside the soccer stadium 22.

[0030] The base station 20 transmits and receives various types of information via radio waves to and from the information processing device 12 and the unmanned aircraft 27. That is, the information processing device 12 is wirelessly communicably connected to the unmanned aircraft 27 via the base station 20. By performing wireless communication with the unmanned aircraft 27 via the base station 20, the information processing device 12 controls the unmanned aircraft 27 and acquires an image obtained by being captured by the imaging device 18 from the unmanned aircraft 27.

[0031] The base station 20 transmits various types of information to the receiver 34 via wireless communication. The information processing device 12 transmits various videos to the receiver 34 via the base station 20. The receiver 34 receives the various videos transmitted from the information processing device 12 and displays the received various videos on the screen 34A. Note that the receiver 34 is used, for example, for viewing by an unspecified number of spectators and the like. The installation location of the receiver 34 may be inside the soccer stadium 22 or outside the soccer stadium 22 (for example, a public viewing venue) or the like. Here, an example form of transmitting various types of information to the receiver 34 via wireless communication is given, but the technology of the present disclosure is not limited to this. For example, it may be a form of transmitting various types of information to the receiver 34 via wired communication.

[0032] The information processing device 12 is a device corresponding to a server, and the smart device 14 is a device corresponding to a client terminal for the information processing device 12. By the information processing device 12 and the smart device 14 performing wireless communication with each other via the base station 20, the smart device 14 requests the information processing device 12 to provide various services, and the information processing device 12 provides the smart device 14 with a service corresponding to the request from the smart device 14.

[0033] As an example, as shown in FIG. 2, the information processing apparatus 12 acquires an aerial view video 46A showing an area including the soccer field 24 when observed from above from the unmanned aerial vehicle 27. The aerial view video 46A is a moving image obtained by imaging the area including the soccer field 24 from above by the imaging device 18 of the unmanned aerial vehicle 27 with the area including the soccer field 24 as an imaging area (hereinafter, also simply referred to as the "imaging area"). Here, although the case where the aerial view video 46A is a moving image is exemplified, the aerial view video 46A is not limited to this, and may be a still image showing the area including the soccer field 24 when observed from above.

[0034] The information processing apparatus 12 acquires an imaging video 46B showing the imaging area when observed from the positions of the respective imaging devices 16 from each of the plurality of imaging devices 16. The imaging video 46B is a moving image obtained by imaging the imaging area by each of the plurality of imaging devices 16. Here, although the case where the imaging video 46B is a moving image is exemplified, the imaging video 46B is not limited to this, and may be a still image showing the imaging area when observed from the positions of the respective imaging devices 16. Here, the aerial view video 46A and the imaging video 46B are examples of the "plurality of images" according to the technology of the present disclosure. Hereinafter, for convenience of explanation, when there is no need to distinguish between the aerial view video 46A and the plurality of imaging videos 46B, they are referred to as "a plurality of original videos" without reference numerals.

[0035] The information processing apparatus 12 generates a virtual viewpoint video 46C based on a plurality of original videos. The virtual viewpoint video 46C is a video showing an imaging area when observing the imaging area from a specific viewpoint position and a specific line-of-sight direction. In the example shown in FIG. 2, the virtual viewpoint video 46C refers to, for example, a virtual viewpoint video showing an imaging area when observing the soccer field 24 at an angle of view 43 from the viewpoint position 42 and the line-of-sight direction 44 within the viewing seat 26. An example of the virtual viewpoint video 46C includes a moving image using 3D polygons. The viewpoint position 42, the line-of-sight direction 44, and the angle of view 43 are not fixed. That is, the viewpoint position 42, the line-of-sight direction 44, and the angle of view 43 vary according to an instruction from the viewer 28 or the like. Here, an example form in which the viewpoint position 42, the line-of-sight direction 44, and the angle of view 43 are not fixed is described, but the technology of the present disclosure is not limited thereto, and the viewpoint position 42, the line-of-sight direction 44, and the angle of view 43 may be fixed.

[0036] The information processing apparatus 12 generates a moving image using 3D polygons by synthesizing a plurality of images (here, as an example, the overhead images that make up the overhead video 46A and the plurality of captured images that make up the plurality of captured videos 46B (hereinafter, when there is no need to distinguish between the overhead images and the plurality of captured images for the sake of convenience of explanation, they are collectively referred to as "a plurality of original images" without symbols)) obtained by capturing an imaging area with a plurality of imaging devices. The information processing apparatus 12 generates a virtual viewpoint video corresponding to the case where the imaging area is observed from an arbitrary position and an arbitrary direction based on the generated moving image using 3D polygons. In the present embodiment, the information processing apparatus 12 generates a virtual viewpoint video 46C showing the subject when observing the subject at the viewpoint position 42, the line-of-sight direction 44, and the angle of view 43. In other words, the virtual viewpoint video 46C refers to a video corresponding to the video obtained by a virtual imaging device (hereinafter also referred to as a "virtual imaging device") installed at the viewpoint position 42 imaging at the angle of view 43 with the line-of-sight direction 44 as the imaging direction. Here, a moving image is exemplified as the virtual viewpoint video 46C, but it is not limited thereto, and a still image may also be used. Note that the viewpoint position 42 is an example of the "viewpoint position" according to the technology of the present disclosure. The line-of-sight direction 44 is an example of the "line-of-sight direction" according to the technology of the present disclosure, and the angle of view 43 is an example of the "angle of view" according to the technology of the present disclosure. Also, the virtual viewpoint video 46C is an example of the "virtual viewpoint image" according to the technology of the present disclosure.

[0037] Also, here, an exemplary form is shown in which the overhead video 46A obtained by being captured by the imaging device 18 is also used for generation, but the technology of the present disclosure is not limited thereto. For example, the overhead video 46A may not be used for generating the virtual viewpoint video 46C, and only the plurality of captured videos 46B obtained by being captured by each of the plurality of imaging devices 16 may be used for generating the virtual viewpoint video 46C. That is, the virtual viewpoint video 46C may be generated only from the videos obtained by being captured by the plurality of imaging devices 16 without using the video obtained from the imaging device 18. Note that if the video obtained from the imaging device 18 (for example, a drone) is used, it becomes possible to generate a virtual viewpoint video with higher accuracy.

[0038] The information processing device 12 selectively transmits the reference video 46D (see FIGS. 6 and 7, for example) and the virtual viewpoint video 46C, which will be described later, as distribution videos to the smart device 14 and the receiver 34. Hereinafter, when it is not necessary to distinguish between the reference video 46D and the virtual viewpoint video 46C for the sake of convenience of explanation, they are also referred to as "distribution videos".

[0039] As shown in FIG. 3 as an example, the information processing device 12 includes a computer 50, a reception device 52, a display 53, a first communication I / F 54, and a second communication I / F 56. The computer 50 includes a CPU 58, a storage 60, and a memory 62, and the C PU 58, the storage 60, and the memory 62 are connected via a bus 64. In the example shown in FIG. 3, for the sake of illustration, one bus is shown as the bus 64, but a plurality of buses may be used. Further, the bus 64 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, etc. Also, the memory 62 may be built in the CPU 58. The CPU 58 is an example of a "processor" according to the technology of the present disclosure, and the storage 60 and the memory 62 are examples of a "memory" according to the technology of the present disclosure.

[0040] The CPU 58 controls the entire information processing device 12. The storage 60 stores various parameters and various programs. The storage 60 is a non-volatile storage device. Here, as an example of the storage 60, a flash memory is adopted, but it is not limited thereto, and an EEPROM, an HDD, an SSD, etc. may be used. The memory 62 is a storage device. Various information is temporarily stored in the memory 62. The memory 62 is used as a work memory by the CPU 58. Here, as an example of the memory 62, a DRAM is adopted, but it is not limited thereto, and other types of storage devices may be used.

[0041] The reception device 52 receives instructions from users of the information processing apparatus 12 or the like. Examples of the reception device 52 include a touch panel, hard keys, and a mouse. The reception device 52 is connected to a bus 64 or the like, and the instructions received by the reception device 52 are acquired by the CPU 58.

[0042] The display 53 is connected to the bus 64 and displays various information under the control of the CPU 58. An example of the display 53 is a liquid crystal display. Note that not limited to the liquid crystal display, other types of displays such as an organic EL display may be adopted as the display 53.

[0043] The first communication I / F 54 is connected to the LAN cable 30. The first communication I / F 54 is realized by, for example, a device having an FPGA. The first communication I / F 54 is connected to the bus 64 and controls the exchange of various information between the CPU 58 and the plurality of imaging devices 16. For example, the first communication I / F 54 controls the plurality of imaging devices 16 according to the request of the CPU 58. In addition, the first communication I / F 54 acquires the captured video 46B (see FIG. 2) obtained by being captured by each of the plurality of imaging devices 16, and outputs the acquired captured video 46B to the CPU 58. Here, the first communication I / F 54 is illustrated as a wired communication I / F, but it may be a wireless communication I / F such as a high-speed wireless LAN.

[0044] The second communication I / F 56 is wirelessly communicably connected to the base station 20. The second communication I / F 56 is realized by, for example, a device having an FPGA. The second communication I / F 56 is connected to the bus 64. The second communication I / F 56 controls the exchange of various information between the CPU 58 and the unmanned aircraft 27 in a wireless communication manner via the base station 20. Also, the second communication I / F 56 controls the exchange of various information between the CPU 58 and the smart device 14 in a wireless communication manner via the base station 20. Further, the second communication I / F 56 controls the transmission of various videos from the CPU 58 to the image receiver 34 in a wireless communication manner via the base station 20. Note that at least one of the first communication I / F 54 and the second communication I / F 56 can be configured with a fixed circuit instead of an FPGA. Also, at least one of the first communication I / F 54 and the second communication I / F 56 may be a circuit configured with an ASIC, an FPGA, and / or a PLD, etc.

[0045] As an example, as shown in FIG. 4, the smart device 14 includes a computer 70, a gyro sensor 74, a reception device 76, a display 78, a microphone 80, a speaker 82, an imaging device 84, and a communication I / F 86. The computer 70 includes a CPU 88, a storage 90, and a memory 92, and the CPU 88, the storage 90, and the memory 92 are connected via a bus 94. In the example shown in FIG. 4, for the sake of illustration, one bus is shown as the bus 94, but the bus 94 may be a plurality of buses. The bus 94 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0046] The CPU 88 controls the entire smart device 14. The storage 90 stores various parameters and various programs. The storage 90 is a non-volatile storage device. Here, as an example of the storage 90, a flash memory is adopted. The flash memory is merely an example, and as the storage 90, for example, instead of the flash memory, or in combination with the flash memory, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory can be mentioned. Also, the non-volatile storage device may be an EEPROM, an HDD, and / or an SSD, etc. The memory 92 temporarily stores various information and is used as a work memory by the CPU 88. As an example of the memory 92, a RAM can be mentioned, but it is not limited to this, and other types of storage devices may also be used.

[0047] The gyro sensor 74 measures the angle around the yaw axis of the smart device 14 (hereinafter also referred to as the "yaw angle"), the angle around the roll axis of the smart device 14 (hereinafter also referred to as the "roll angle"), and the angle around the pitch axis of the smart device 14 (hereinafter also referred to as the "pitch angle"). The gyro sensor 74 is connected to the bus 94, and the angle information indicating the yaw angle, roll angle, and pitch angle measured by the gyro sensor 74 is acquired by the CPU 88 via the bus 94 or the like.

[0048] The reception device 76 receives an instruction from a user or the like of the smart device 14 (here, as an example, the viewer 28). As an example of the reception device 76, a touch panel 76A and hard keys, etc. can be mentioned. The reception device 76 is connected to the bus 94, and the instruction received by the reception device 76 is acquired by the CPU 88.

[0049] The display 78 is connected to the bus 94 and displays various information under the control of the CPU 88. As an example of the display 78, a liquid crystal display can be mentioned. Note that not limited to the liquid crystal display, other types of displays such as an organic EL display may be adopted as the display 78.

[0050] The smart device 14 is provided with a touch panel display, which is realized by a touch panel 76A and a display 78. That is, the touch panel display is formed by overlapping the touch panel 76A on the display area of the display 78, or by incorporating a touch panel function inside the display 78 (the "in-cell" type).

[0051] The microphone 80 converts the collected sound into an electrical signal. The microphone 80 is connected to the bus 94. The electrical signal obtained by converting the sound collected by the microphone 80 is acquired by the CPU 88 via the bus 94.

[0052] The speaker 82 converts an electrical signal into sound. The speaker 82 is connected to the bus 94. The speaker 82 receives the electrical signal output from the CPU 88 via the bus 94, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the smart device 14.

[0053] The imaging device 84 acquires an image indicating a subject by imaging the subject. The imaging device 84 is connected to the bus 94. The image obtained by imaging the subject by the imaging device 84 is acquired by the CPU 88 via the bus 94.

[0054] The communication I / F 86 is wirelessly communicably connected to the base station 20. The communication I / F 86 is realized by, for example, a device composed of a circuit (such as an ASIC, FPGA, and / or PLD, etc.). The communication I / F 86 is connected to the bus 94. The communication I / F 86 controls the exchange of various information between the CPU 88 and an external device in a wireless communication manner via the base station 20. Here, examples of the "external device" include the information processing device 12.

[0055] As an example, as shown in FIG. 5, in the information processing apparatus 12, the storage 60 stores a reference video generation processing program 60A, a virtual viewpoint video generation processing program 60B, and a video distribution processing program 60C. The virtual viewpoint video generation processing program 60B is an example of the "program" according to the technology of the present disclosure. In the following, when it is not necessary to distinguish between the reference video generation processing program 60A, the virtual viewpoint video generation processing program 60B, and the video distribution processing program 60C, they are collectively referred to as the "information processing apparatus side program" without reference numerals.

[0056] The CPU 58 reads out the information processing apparatus side program from the storage 60 and executes the read information processing apparatus side program on the memory 62. The CPU 58 exchanges various information with the smart device 14, the imaging device 16, and the unmanned aircraft 27 according to the information processing apparatus side program executed on the memory 62, and transmits various videos to the receiver 34.

[0057] The CPU 58 reads out the reference video generation processing program 60A from the storage 60 and executes the read reference video generation processing program 60A on the memory 62. The CPU 58 operates as the control unit 100 and the reference video generation unit 102 according to the reference video generation processing program 60A executed on the memory 62. By operating as the control unit 100 and the reference video generation unit 102, the CPU 58 executes the reference video generation processing (see FIG. 17) described later.

[0058] The CPU 58 reads out the virtual viewpoint video generation processing program 60B from the storage 60 and executes the read virtual viewpoint video generation processing program 60B on the memory 62. The CPU 58 operates as the control unit 100 and the virtual viewpoint video generation unit 104 according to the virtual viewpoint video generation processing program 60B executed on the memory 62, and thereby executes the virtual viewpoint video generation processing (see FIG. 18) described later.

[0059] The CPU 58 reads the video distribution processing program 60C from the storage 60 and executes the read video distribution processing program 60C on the memory 62. The CPU 58 operates as the control unit 100 according to the video distribution processing program 60C executed on the memory 62. By operating as the control unit 100, the CPU 58 executes the video distribution processing (see FIG. 19) described later.

[0060] An example of the content of the reference video generation process will be described with reference to FIG. 6. A plurality of original videos obtained by being captured by a plurality of imaging devices are stored in the memory 62 of the information processing apparatus 12. Each of the plurality of original videos is a moving image obtained by being captured at a predetermined frame rate (for example, 30 fps) and consists of still images of a plurality of frames. In the memory 62, for each of the plurality of original videos, a region for storing still images of a predetermined number of frames (for example, several thousand frames) is provided, and the old still images are overwritten with the newly acquired still images. As a result, the memory 62 stores a plurality of original videos for a predetermined period of time.

[0061] In the reference video generation process, in the information processing apparatus 12, an instruction from a video producer (not shown) is received by the reception device 52, and the reference imaging device identifier 35 is output by the reception device 52 according to the received instruction. The reference imaging device identifier 35 is an identifier that specifies one of the plurality of imaging devices.

[0062] Specifically, in a state where a plurality of original videos are arranged and displayed on the display 53, one video is specified from the plurality of original videos by a video producer via the touch panel. Then, an identifier associated with the imaging device 16 or 18 that captured the specified captured video 46B or the aerial view video 46A is output from the reception device 52 as the reference imaging device identifier 35. Hereinafter, for convenience of explanation, the imaging device 16 used for imaging to obtain the specified captured video 46B, or the imaging device 18 used for imaging to obtain the specified aerial view video 46A is referred to as the reference imaging device 17. That is, the reference imaging device 17 is one of the plurality of imaging devices. Here, the reference imaging device 17 is an example of the "reference imaging device" according to the technology of the present disclosure.

[0063] The position, imaging direction, and field of view angle of the reference imaging device 17 can be changed. Also, the reference imaging device 17 can be switched among the plurality of imaging devices. Here, a case where the position, imaging direction, and field of view angle of the reference imaging device 17 can be changed is illustrated, but the technology of the present disclosure is not limited to this, and any one or any two of the position, imaging direction, and field of view angle of the reference imaging device 17 may be changeable. Also, here, a case where one of the plurality of imaging devices is specified as the reference imaging device 17 is illustrated, but it is not limited to this, and an imaging device different from the plurality of imaging devices may be specified as the reference imaging device 17. In this case, the imaging device specified as the reference imaging device 17 may be a movable video camera (for example, a camera for television relay) whose position, imaging direction, and field of view angle are operated by a cameraman.

[0064] In the example shown in FIG. 6, one of the plurality of imaging devices 16 is designated as the reference imaging device 17. The reference video generation unit 102 reads out the captured video 46B obtained by the reference imaging device 17 capturing the soccer field 24 from the memory 62, and superimposes information indicating the game situation (for example, score, player information, and / or remaining time of the game) on the read captured video 46B to generate a reference video 46D. Here, the "reference video 46D" is, for example, a live relay video. The reference video 46D is an example of the "first reference image", "second reference image", "third reference image", and "fourth reference image" according to the technology of the present disclosure.

[0065] Note that here, the reference video 46D is generated based on the captured video 46B obtained by being captured by the imaging device 16 designated by the reference imaging device identifier 35, but the captured video 46B itself may be adopted as the reference video 46D. Also, instead of the imaging device 16, the imaging device 18 may be designated as the reference imaging device 17. In this case, the reference video 46D is generated by the reference video generation unit 102 based on the bird's-eye view video 46A.

[0066] The reference video generation unit 102 stores the generated reference video 46D in the memory 62. Here, the reference video 46D is a moving image that can be displayed at a predetermined frame rate (for example, 30 fps). The control unit 100 reads out the reference video 46D from the memory and outputs it to the receiver 34 via the second communication I / F 56 and the base station 20. Also, when the control unit 100 has not received a virtual viewpoint video generation instruction 107 (see FIGS. 7 and 8) described later, the control unit 100 also outputs the reference video 46D to the smart device 14. In this case, the reference video 46D is displayed on the display 78 (see FIG. 8).

[0067] Next, an example of the content of the virtual viewpoint video generation process will be described with reference to FIG. 7. The touch panel 76A of the smart device 14 receives a virtual viewpoint video generation instruction 107 (see FIG. 8) from the viewer 28. The control unit 100 is received by the touch panel 76A Obtain a virtual viewpoint video generation instruction 107. The virtual viewpoint video generation instruction 107 is, for example, turned on via the touch panel 76A by the instruction button 106 (see FIG. 8) displayed on the display 78 by the viewer 28 who is viewing the reference video 46D on the display 78 of the smart device 14. Note that the virtual viewpoint video generation instruction 107 is an example of the "instruction" according to the technology of the present disclosure.

[0068] The virtual viewpoint video generation instruction 107 is an instruction to start generating a virtual viewpoint video 46C based on a plurality of original videos obtained by imaging the soccer field 24 with a plurality of imaging devices. For example, when the virtual viewpoint video generation instruction 107 instructs the information processing device 12 to start generating the virtual viewpoint video 46C and to display the generated virtual viewpoint video 46C on the display 78, it is given to the smart device 14 by the viewer 28.

[0069] The control unit 100 obtains reference imaging device information 46E. The reference imaging device information 46E is the position, imaging direction, and field of view angle of the reference imaging device 17. Here, the position, imaging direction, and field of view angle of the reference imaging device 17 are exemplified as the reference imaging device information 46E, but the technology of the present disclosure is not limited to this, and it may be a position close to the position of the reference imaging device 17, a direction close to the imaging direction of the reference imaging device 17, and a field of view angle close to the field of view angle of the reference imaging device 17, and any information corresponding to the position, imaging direction, and field of view angle of the reference imaging device 17 may be used.

[0070] A plurality of imaging devices continuously transmit imaging device information corresponding to their respective positions, imaging directions, and field angles in the soccer field 24. The control unit 100 receives the imaging device information transmitted by the plurality of imaging devices and stores it in the memory 62. Further, the control unit 100 acquires the imaging device information transmitted by a designated reference imaging device 17 among the plurality of imaging devices as reference imaging device information 46E by reading it from the memory 62. Note that the reference imaging device information 46E is an example of the "reference imaging device information" according to the technology of the present disclosure. Also, the reference imaging device 17 is an example of the "transmission device" according to the technology of the present disclosure. Here, "continuously transmitting imaging device information" means that the imaging device information is transmitted multiple times. The transmission interval of the imaging device information may or may not be constant.

[0071] The control unit 100 activates the extraction unit 108. The extraction unit 108 reads the reference video 46D from the memory 62 and acquires image quality information 46F indicating the image quality of the reference video 46D. The image quality information 46F acquired by the extraction unit 108 is stored in the memory 62. The image quality information 46F includes, for example, the resolution, brightness, and contrast of the image. Note that the image quality information 46F is an example of the "image quality information" according to the technology of the present disclosure.

[0072] Furthermore, the control unit 100 activates the virtual viewpoint video generation unit 104 on the condition that a virtual viewpoint video generation instruction 107 is given. The control unit 100 reads the captured video 46B, the bird's-eye view video 46A, the image quality information 46F, and the reference imaging device information 46E from the memory 62 and outputs them to the virtual viewpoint video generation unit 104.

[0073] The virtual viewpoint video generation unit 104 generates a virtual viewpoint video 46C using 3D polygons based on the reference imaging device information 46E by synthesizing a plurality of original images that constitute a plurality of original videos on the condition that a virtual viewpoint video generation instruction 107 is given. The virtual viewpoint video 46C is a video corresponding to the reference video 46D. For example, the virtual viewpoint video 46C is generated as a virtual viewpoint video 46C by imaging the soccer field 24 from the viewpoint position 42, the line-of-sight direction 44, and the angle of view that are the same as the position, the imaging direction, and the angle of view included in the reference imaging device information 46E. That is, the virtual imaging device coincides with the reference imaging device 17, and the viewpoint position 42, the line-of-sight direction 44, and the angle of view 43 of the virtual viewpoint video 46C coincide with the position, the imaging direction, and the angle of view of the reference imaging device 1 7.

[0074] Also, the virtual viewpoint video generation unit 104 determines the image quality of the virtual viewpoint video 46C to be generated based on the image quality information 46F on the condition that the virtual viewpoint video generation instruction 107 is given. For example, the virtual viewpoint video generation unit 104 makes the resolution, the brightness, and the contrast of the virtual viewpoint video 46C to be generated coincide with the resolution, the brightness, and the contrast included in the image quality information 46F. Here, a form example in which the resolution, the brightness, and the contrast of the virtual viewpoint video 46C are made to coincide with the resolution, the brightness, and the contrast included in the image quality information 46F is described, but the technology of the present disclosure is not limited to this, and the resolution, the brightness, and the contrast of the virtual viewpoint video 46C may be made to approach the resolution, the brightness, and the contrast included in the image quality information 46F. The virtual viewpoint video generation unit 104 stores the generated virtual viewpoint video 46C in the memory 62.

[0075] Next, a mode of displaying the virtual viewpoint video 46C on the display 78 of the smart device 14 will be described. As shown in FIG. 8 as an example, while the reference video 46D is displayed on the display 78, an instruction button 106 is displayed at the upper part of the display 78. When the viewer 28 touches the instruction button 106 from above the touch panel 76A, the virtual viewpoint video generation instruction 107 is acquired from the smart device 14 by the information processing device 12.

[0076] The control unit 100 activates the virtual viewpoint video generation unit 104 to generate the virtual viewpoint video 46C as described above on the condition that the virtual viewpoint video generation instruction 107 is given. The control unit 100 reads out from the memory 62 the reference video 46D obtained by the reference imaging device 17 and the virtual viewpoint video 46C generated based on the reference imaging device information 46E. The control unit 100 generates a superimposed video 46G obtained by superimposing the reference video 46D and the virtual viewpoint video 46C. Specifically, the superimposed video 46G is a video in which the virtual viewpoint video 46C is superimposed on the reference video 46D. The control unit 100 outputs the superimposed video 46G to the smart device 14 via the second communication I / F 56 and the base station 20. The superimposed video 46G is displayed on the display 78 of the smart device 14. Note that the superimposed video 46G is an example of the "superimposed image" according to the technology of the present disclosure.

[0077] Also, the control unit 100 gradually changes the ratio between the reference video 46D and the virtual viewpoint video 46C in the superimposed video 46G. As an example, as shown in FIG. 9, after the virtual viewpoint video generation instruction 107 is received, the control unit 100 generates a superimposed image by superimposing the reference image and the virtual viewpoint image at a ratio of, for example, 9:1, and outputs it to the smart device 14. Here, the ratio refers to, for example, the ratio between the average pixel value of the reference image in the superimposed video 46G and the average pixel value of the virtual viewpoint image in the superimposed video 46G. In other words, here, the ratio refers to the ratio of the degree to which the reference image occupies and the degree to which the virtual viewpoint image occupies in terms of pixels of the superimposed video 46G. Also, here, the reference image refers to an image for one frame constituting the reference video 46D. The virtual viewpoint image refers to an image for one frame constituting the virtual viewpoint video 46C. Note that the ratio is an example of the "ratio" according to the technology of the present disclosure.

[0078] The superimposed image generated by the control unit 100 according to a ratio of 9:1 is output to the smart device 14. For example, when the output frame rates of the superimposed video 46G and the virtual viewpoint video 46C to the smart device 14 are 30 fps, the control unit 100 outputs the superimposed image to the smart device 14 while changing the ratio in units of 3 frames. For example, first, the control unit 100 outputs the superimposed images for 3 frames generated at a ratio of 9:1 to the smart device 14.

[0079] Subsequently, the control unit 100 generates superimposed images for 3 frames by superimposing the reference image and the virtual viewpoint image at a ratio of 8:2 and outputs them to the smart device 14. Thereafter, the control unit 100 gradually changes the ratio of the reference image and the virtual viewpoint image to 7:3, 6:4, ···, 1:9 and outputs the generated superimposed images to the smart device 14 in units of 3 frames each. After that, the control unit 100 outputs the virtual viewpoint video 46C composed of the virtual viewpoint images to the smart device 14.

[0080] For the superimposition of the reference image and the virtual viewpoint image, for example, alpha blending is applied. Alpha blending is a technique for synthesizing two images according to a coefficient (alpha value). The alpha value is a value in the range from 0 to 1. An image with the alpha value set to 1 is completely opaque, and an image with the alpha value set to 0 is completely transparent. Therefore, when the reference image and the virtual viewpoint image are superimposed at a ratio of 9:1, for example, the alpha value of the reference image is set to 0.9 and the alpha value of the virtual viewpoint image is set to 0.1.

[0081] In the example shown in FIG. 9, the smart device 14 depicted at the upper part of FIG. 9 is in a state at the timing when the instruction button 106 is touched, and on the display 78, a superimposed image generated by superimposing the reference image and the virtual viewpoint image at a ratio of 9:1 is being displayed. The smart device 14 depicted in the center of FIG. 9 is in a state at the timing when, for example, 0.5 seconds have elapsed since the instruction button 106 was touched, and on the display 78, a superimposed image generated by superimposing the reference image and the virtual viewpoint image at a ratio of 5:5 is being displayed. The smart device 14 depicted at the lower part of FIG. 9 is in a state at the timing when, for example, 1.0 second has elapsed since the instruction button 106 was touched, and the display of the display 78 has switched from the superimposed image to the virtual viewpoint image. Thus, the display of the smart device 14 gradually changes from the reference video 46D to the virtual viewpoint video 46C via the superimposed video over 1 second.

[0082] When the virtual viewpoint video 46C is displayed on the display 78, the instruction button 106 is displayed at the upper part of the display 78. The instruction button 106 is a button that is operated when changing the display of the display 78 from the virtual viewpoint video 46C to the reference video 46D. When the viewer 28 touches the instruction button 106 from above on the touch panel 76A, the reference video generation instruction 109 is acquired by the information processing apparatus 12 from the smart device 14. The control unit 100 stops the generation and output of the virtual viewpoint video 46C and outputs the reference video 46D to the smart device 14 on the condition that the reference video generation instruction 109 is given.

[0083] As shown in FIG. 10 as an example, the control unit 100 outputs the virtual viewpoint video 46C to the display 78 of the smart device 14 and receives a change signal 120 that continuously changes at least one of the viewpoint position 42, the line-of-sight direction 44, and the field angle 43 in the output virtual viewpoint video 46C. Here, the display 78 is an example of the "display" according to the technology of the present disclosure. The change signal 120 is an example of the "change signal" according to the technology of the present disclosure.

[0084] In the virtual viewpoint video 46C displayed on the display 78, a change instruction 110 from the viewer 28 is received by the touch panel 76A. The change instruction 110 is roughly classified into a viewpoint position change instruction 110A for changing the viewpoint position 42, a line-of-sight direction change instruction 110B for changing the line-of-sight direction 44, and an angle-of-view change instruction 110C for changing the angle of view 43. The change instruction 110 received by the touch panel 76A is output from the smart device 14 to the control unit 100.

[0085] The control unit 100 generates a change signal 120 indicating the change instruction 110 input from the smart device 14. The change signal 120 is roughly classified into a viewpoint position change signal 120A indicating the viewpoint position change instruction 110A, a line-of-sight direction change signal 120B indicating the line-of-sight direction change instruction 110B, and an angle-of-view change signal 120C indicating the angle-of-view change instruction 110C. The change signal 120 is output from the control unit 100 to the virtual viewpoint video generation unit 104.

[0086] When the virtual viewpoint video generation unit 104 receives the viewpoint position change signal 120A from the control unit 100 it executes a viewpoint position change process 104A for changing the viewpoint position 42. When the virtual viewpoint video generation unit 104 receives the line-of-sight direction change signal 120B from the control unit 100, it executes a line-of-sight direction change process 104B for changing the line-of-sight direction 44. When the virtual viewpoint video generation unit 104 receives the angle-of-view change signal 120C from the control unit 100, it executes an angle-of-view change process 104C for changing the angle of view 43.

[0087] As an example, as shown in FIG. 11, the viewpoint position change instruction 110A is an instruction for changing the viewpoint position 42 of the virtual viewpoint video 46C. The viewpoint position change instruction 110A is received by the touch panel 76A when the viewer 28 linearly slides a finger on the touch panel 76A. For example, as shown in the smart device 14 depicted at the top of FIG. 11, when the viewer 28 slides a finger in the direction of the dotted arrow on the touch panel 76A, the viewpoint position change instruction 110A is given to the smart device 14.

[0088] The virtual viewpoint video 46C displayed on the smart device 14 depicted at the upper part of FIG. 11 is a video captured by a virtual imaging device disposed at the viewpoint position 42A. When moving the viewpoint position 42A of the virtual imaging device to the viewpoint position 42B, the viewer 28 gives a viewpoint position change instruction 110A to the smart device 14 by performing a swipe operation of continuously sliding a finger on the touch panel 76A in a direction opposite to the moving direction of the viewpoint position.

[0089] Thus, when the viewpoint position change instruction 110A is given to the smart device 14, as described above, the viewpoint position change process 104A is performed by the virtual viewpoint video generation unit 104. By performing the viewpoint position change process 104A by the virtual viewpoint video generation unit 104, as shown in FIG. 12 as an example, the virtual imaging device moves from the viewpoint position 42A to the viewpoint position 42B. The virtual viewpoint video 46C generated based on the viewpoint position 42B by the virtual viewpoint video generation unit 104 is obtained by being virtually captured by a virtual imaging device disposed at the viewpoint position 42B.

[0090] As shown in FIG. 13 as an example, the viewer 28 can also give a line-of-sight direction change instruction 110B to the smart device 14 together with the viewpoint position change instruction 110A. The line-of-sight direction change instruction 110B is an instruction for changing the line-of-sight direction 44 of the virtual viewpoint video 46C. In order to give the line-of-sight direction change instruction 110B to the smart device 14, for example, the viewer 28 slides a finger curvilinearly on the touch panel 76A. As shown at the upper part of FIG. 13, when the viewer 28 slides a finger in the direction of the dotted arrow on the touch panel 76A, the viewpoint position change instruction 110A is received by the touch panel 76A.

[0091] The virtual viewpoint video 46C displayed on the smart device 14 depicted at the top of FIG. 13 is a video obtained by imaging the line-of-sight direction 44C with a virtual imaging device arranged at the viewpoint position 42C. When moving the viewpoint position 42C of the virtual imaging device to the viewpoint position 42D and changing the line-of-sight direction 44C of the virtual imaging device to the line-of-sight direction 44D, the viewer 28 gives a viewpoint position change instruction 110A and a line-of-sight direction change instruction 110B to the smart device 14 by means of a swipe operation of continuously sliding a finger on the touch panel 76A in a direction opposite to the moving direction of the viewpoint position.

[0092] In this way, when the viewpoint position change instruction 110A and the line-of-sight direction change instruction 110B are given to the smart device 14, the viewpoint position change process 104A is performed by the virtual viewpoint video generation unit 104, and the line-of-sight direction change process 104B is performed. As a result, as shown in FIG. 14 as an example, the viewpoint position of the virtual imaging device is changed from the viewpoint position 42C to the viewpoint position 42D, and the line-of-sight direction of the virtual imaging device is changed from the line-of-sight direction 44C to the line-of-sight direction 44D. The virtual viewpoint video 46C generated by the virtual viewpoint video generation unit 104 based on the viewpoint position 42D and the line-of-sight direction 44D is obtained by virtually imaging the line-of-sight direction 44D with a virtual imaging device arranged at the viewpoint position 42D. direction 44D is obtained by virtual imaging.

[0093] As shown in FIGS. 15 and 16 as an example, the field-of-view angle change instruction 110C is an instruction for changing the field-of-view angle 43 of the virtual viewpoint video 46C. In order to give the field-of-view angle change instruction 110C to the smart device 14, the viewer 28 performs a pinch-in operation of pinching the screen with two fingers or a pinch-out operation of releasing the pinched state of the screen on the touch panel 76A. The pinch-out operation is performed when narrowing the field-of-view angle, and the pinch-in operation is performed when widening the field-of-view angle.

[0094] In the example shown in FIG. 15, on the touch panel 76A on which the virtual viewpoint video 46C is displayed, a pinch-out operation is performed by the viewer 28. When the pinch-out operation is performed on the touch panel 76A, an instruction to narrow the viewing angle is given to the smart device 14 as the viewing angle change instruction 110C. In this way, when an instruction to narrow the viewing angle is given to the smart device 14 as the viewing angle change instruction 110C, as described above, the viewing angle change process 104C corresponding to the pinch-out operation is performed by the virtual viewpoint video generation unit 104. When the viewing angle change process 104C corresponding to the pinch-out operation is performed by the virtual viewpoint video generation unit 104, as shown in FIG. 15 as an example, the virtual viewpoint video 46C is generated by the virtual viewpoint video generation unit 104 so that the viewing angle becomes narrower according to the pinch-out operation.

[0095] In the example shown in FIG. 16, on the touch panel 76A on which the virtual viewpoint video 46C is displayed, a pinch-in operation is performed by the viewer 28. When the pinch-in operation is performed on the touch panel 76A, an instruction to widen the viewing angle is given to the smart device 14 as the viewing angle change instruction 110C. In this way, when an instruction to widen the viewing angle is given to the smart device 14 as the viewing angle change instruction 110C, as described above, the viewing angle change process 104C corresponding to the pinch-in operation is performed by the virtual viewpoint video generation unit 104. When the viewing angle change process 104C corresponding to the pinch-in operation is performed by the virtual viewpoint video generation unit 104, as shown in FIG. 16 as an example, the virtual viewpoint video 46C is generated by the virtual viewpoint video generation unit 104 so that the viewing angle becomes wider according to the pinch-in operation.

[0096] Next, an example of the flow of the reference video generation process, the virtual viewpoint video generation process, and the video distribution process performed by the information processing device 12 will be described with reference to FIGS. 17 to 19.

[0097] The reference video generation process shown in FIG. 17 as an example is executed by the CPU 58 according to the reference video generation process program 60A when an instruction to execute the reference video generation process is received by the reception device 52.

[0098] In the reference video generation process shown in FIG. 17, first, in step S10, the control unit 100 determines whether the image generation timing has arrived. As an example of the image generation timing, there is a timing defined according to the frame rate. For example, when the frame rate of the information processing apparatus 12 is 30 fps, the image generation timing arrives every 1 / 30 second. In step S10, if the image generation timing has not arrived, the determination is negative, and the reference video generation process proceeds to step S15. In step S10, if the image generation timing has arrived, the determination is affirmative, and the reference video generation process proceeds to step S11.

[0099] In step S11, the control unit 100 acquires the reference imaging device identifier 35. The reference imaging device identifier 35 is an identifier that designates one of a plurality of imaging devices. For example, it is received irregularly when the reception device 52 is operated by a video producer and is overwritten and stored in the memory 62. The control unit 100 acquires the reference imaging device identifier 35 received at the closest timing from the memory 62. Thereafter, the reference video generation process proceeds to step S12.

[0100] In step S12, the control unit 100 reads out an imaging image or an aerial view image for one frame obtained by imaging with the imaging device specified by the reference imaging device identifier 35, that is, the reference imaging device 17, from the memory 62 and outputs it to the reference video generation unit 102. Thereafter, the reference video generation process proceeds to step S13.

[0101] In step S13, the reference video generation unit 102 superimposes the game situation information on the imaging image or the aerial view image obtained by imaging with the reference imaging device 17 to generate a reference image for one frame. The game situation information includes, for example, scores, player information, and / or the remaining time of the game. Thereafter, the reference video generation process proceeds to step S14.

[0102] In step S14, the reference video generation unit 102 stores the reference image for one frame generated in step S13 in the memory 62. After that, the reference video generation process proceeds to step S15.

[0103] In step S15, the control unit 100 determines whether a condition for ending the reference video generation process (hereinafter referred to as the "reference video generation process end condition") is satisfied. As an example of the reference video generation process end condition, there is a condition that an instruction to end the generation of the reference video 46D is received by the reception device 52 (see FIG. 3). In step S15, if the reference video generation process end condition is not satisfied, the determination is negative, and the reference video generation process proceeds to step S10. In step S15, if the reference video generation process end condition is satisfied, the determination is positive, and the reference video generation process ends.

[0104] As an example, the virtual viewpoint video generation process shown in FIG. 18 is executed by the CPU 58 according to the virtual viewpoint video generation program 60B when a virtual viewpoint video generation instruction 107 is received by the touch panel 76A of the smart device 14.

[0105] In the virtual viewpoint video generation process shown in FIG. 18, first, in step S20, the control unit 100 determines whether a virtual viewpoint video generation instruction 107 is received by the touch panel 76A of the smart device 14. In step S20, if the virtual viewpoint video generation instruction 107 is not received by the touch panel 76A of the smart device 14, the determination is negative, and the determination in step S20 is made again. In step S20, if the virtual viewpoint video generation instruction 107 is received by the touch panel 76A of the smart device 14, the determination is positive, and the virtual viewpoint video generation process proceeds to step S21. Note that step S20 may be a step after step S22, and specifically, it may be inserted between step S22 and step S23.

[0106] In step S21, the control unit 100 reads the reference imaging device information 46E from the memory 62. The reference imaging device information 46E is information corresponding to the position, imaging direction, and field angle of the reference imaging device 17. Thereafter, the virtual viewpoint video generation process proceeds to step S22.

[0107] In step S22, the control unit 100 determines the viewpoint position 42, line-of-sight direction 44, and field angle 43 of the virtual imaging device based on the reference imaging device information 46E. For example, the control unit 100 determines the viewpoint position 42, line-of-sight direction 44, and field angle 43 of the virtual imaging device so that the field of view of the image obtained by the virtual imaging device is the same as the field of view of the image obtained by the reference imaging device 17. Thereafter, the reference video generation process proceeds to step S23.

[0108] In step S23, the control unit 100 determines whether the image generation timing has arrived. In step S23, if the image generation timing has not arrived, the determination is negative and the virtual viewpoint video generation process proceeds to step S29. In step S23, if the image generation timing has arrived, the determination is affirmative and the virtual viewpoint video generation process proceeds to step S24.

[0109] In step S24, the control unit 100 reads a plurality of original images from the memory 62 based on the viewpoint position 42, line-of-sight direction 44, and field angle 43 determined in step S22. That is, the control unit 100 reads from the memory 62 the plurality of original images required to generate the virtual viewpoint video 46C of the field of view defined by the viewpoint position 42, line-of-sight direction 44, and field angle 43 and outputs them to the virtual viewpoint video generation unit 104. Thereafter, the virtual viewpoint video generation process proceeds to step S25.

[0110] In step S25, the virtual viewpoint video generation unit 104 generates a virtual viewpoint image for one frame of the field of view defined by the viewpoint position 42, the line-of-sight direction 44, and the viewing angle 43 determined in step S22 from a plurality of original images. That is, the virtual viewpoint video generation unit 104 generates a virtual viewpoint image having the same field of view as the reference image. After that, the virtual viewpoint video generation process proceeds to step S26.

[0111] In step S26, the virtual viewpoint video generation unit 104 stores the virtual viewpoint image for one frame generated in step S25 in the memory 62. After that, the virtual viewpoint video generation process proceeds to step S27.

[0112] In step S27, the control unit 100 determines whether a change instruction 110 has been received by the smart device 14. In step S27, if the change instruction 110 has not been received by the smart device 14, the determination is negative, and the virtual viewpoint video generation process proceeds to step S29. In step S27, if the change instruction 110 has been received by the smart device 14, the determination is positive, and the virtual viewpoint video generation process proceeds to step S28.

[0113] In step S28, the control unit 100 determines the viewpoint position 42, the line-of-sight direction 44, and the viewing angle 43 of the virtual imaging device based on the change instruction 110. That is, the viewpoint position 42, the line-of-sight direction 44, and the viewing angle 43 determined in step S22 are discarded, and the viewpoint position 42, the line-of-sight direction 44, and the viewing angle 43 determined in step S28 become valid. That is, in step S25, a virtual viewpoint image for one frame of the field of view defined by the viewpoint position 42, the line-of-sight direction 44, and the viewing angle 43 determined in step S28 is generated. After the process of step S28 is executed, the virtual viewpoint video generation process proceeds to step S29.

[0114] In step S29, the control unit 100 determines whether a condition for ending the virtual viewpoint video generation process (hereinafter referred to as the "virtual viewpoint video generation process end condition") is satisfied. As an example of the virtual viewpoint video generation process end condition, there is a condition that an instruction to end the generation of the virtual viewpoint video 46C is received by the reception device 52 (see FIG. 3). In step S29, if the virtual viewpoint video generation process end condition is not satisfied, the determination is negative, and the virtual viewpoint video generation process proceeds to step S23. In step S29, if the virtual viewpoint video generation process end condition is satisfied, the determination is affirmative, and the virtual viewpoint video generation process ends.

[0115] As an example, the video distribution process shown in FIG. 19 is executed by the CPU 58 according to the video distribution process program 60C when an instruction to start the execution of the video distribution process is received by the touch panel 76A of the smart device 14.

[0116] In the video distribution process shown in FIG. 19, first, in step S31, the control unit 100 determines whether the image output timing has arrived. As an example of the image output timing, there is timing divided at time intervals defined by the output frame rate. In step S31, if the image output timing has not arrived, the determination is negative, and the determination in step S31 is made again. In step S31, if the image output timing has arrived, the determination is affirmative, and the video distribution process proceeds to step S32.

[0117] In step S32, the control unit 100 reads out a reference image for one frame from the memory 62 and outputs it to the smart device 14. Thereafter, the video distribution process proceeds to step S33.

[0118] In step S33, the control unit 100 determines whether a condition for ending the video distribution process (hereinafter referred to as the "video distribution process end condition") is satisfied. As an example of the video distribution process end condition, there is a condition that an instruction to end the video distribution process is received by the reception device 52 or 76. In step S33, if the video distribution process end condition is not satisfied, the determination is negative, and the video distribution process proceeds to step S34. In step S33, if the video distribution process end condition is satisfied, the determination is affirmative, and the video distribution process ends.

[0119] In step S34, the control unit 100 determines whether a virtual viewpoint video generation instruction 107 is received by the touch panel 76A of the smart device 14. In step S34, if the virtual viewpoint video generation instruction 107 is not received by the touch panel 76A of the smart device 14, the determination is negative, and the video distribution process proceeds to step S31. In step S34, if the virtual viewpoint video generation instruction 107 is received by the touch panel 76A of the smart device 14, the determination is affirmative, and the video distribution process proceeds to step S35.

[0120] In step S35, the control unit 100 determines whether the image output timing has arrived. In step S35, if the image output timing has not arrived, the determination is negative, and the determination in step S35 is made again. In step S35, if the image output timing has arrived, the determination is affirmative, and the video distribution process proceeds to step S36.

[0121] In step S36, the control unit 100 reads out a virtual viewpoint image for one frame from the memory 62. Thereafter, the video distribution process proceeds to step S37.

[0122] In step S37, the control unit 100 determines whether the next display frame is an initial display frame shortly after receiving the virtual viewpoint video generation instruction 107. For example, if the next frame is the Nth display frame (N is a natural number greater than or equal to 1) and is smaller than a specific number of frames (e.g., 30) (N < 30), the determination is affirmative, and the video distribution process proceeds to step S38. On the other hand, if the next display frame is the Nth display frame and is greater than or equal to the specific number of frames (N ≥ 30), the determination is negative, and the video distribution process proceeds to step S39. Although the specific number of frames is described as "30", the technology of the present disclosure is not limited thereto, and the specific number of frames may be set to any number among natural numbers.

[0123] In step S38, the control unit 100 generates a superimposed image by superimposing the virtual viewpoint image on the reference image, and outputs the generated superimposed image to the smart device 14. Thereafter, the video distribution process proceeds to step S40.

[0124] In step S39, the control unit 100 outputs the virtual viewpoint image for one frame read in step S36 to the smart device 14. Thereafter, the video distribution process proceeds to step S40 and proceeds.

[0125] In step S40, the control unit 100 determines whether a condition for ending the video distribution process (hereinafter referred to as the "video distribution process end condition") is satisfied. An example of the video distribution process end condition is a condition that the video distribution application has ended. In step S40, if the video distribution process end condition is not satisfied, the determination is negative, and the video distribution process proceeds to step S41. In step S40, if the video distribution process end condition is satisfied, the determination is affirmative, and the video distribution process ends.

[0126] In step S41, the control unit 100 determines whether or not the reference video generation instruction 109 has been accepted by the touch panel 76A of the smart device 14. In step S41, if the reference video generation instruction 109 has not been accepted by the touch panel 76A of the smart device 14, the determination is negative, and the video distribution process proceeds to step S35. In step S41, if the reference video generation instruction 109 has been accepted by the touch panel 76A of the smart device 14, the determination is positive, and the video distribution process proceeds to step S31.

[0127] As described above, the information processing device 12 includes a CPU 58 and a memory 62 connected to or built into the CPU 58. The CPU 58 acquires reference imaging device information 46E corresponding to the position, imaging direction, and angle of view of the reference imaging device 17. Furthermore, the CPU 58 generates the virtual viewpoint video 46C based on the reference imaging device information 46E on the condition that a virtual viewpoint video generation instruction 107, which is an instruction to start generating a virtual viewpoint video 46C based on a plurality of original videos, is given. Therefore, a virtual viewpoint video 46C having the same or similar field of view as the reference imaging device information 46E is generated. According to this configuration, it is possible to reduce the difference in the field of view between the reference video 46D and the virtual viewpoint video 46C, compared to a case where the virtual viewpoint video 46C is generated based on the position, imaging direction, and angle of view of the imaging device 16 or 18, which is different from the reference imaging device 17, among the plurality of imaging devices.

[0128] Also, when the reference imaging device information 46E is continuously transmitted by the reference imaging device 17, the CPU 58 generates the virtual viewpoint video 46C based on the reference imaging device information 46E transmitted from the reference imaging device 17 on the condition that the virtual viewpoint video generation instruction 107 is given. Since the reference imaging device 17 continuously transmits the reference imaging device information 46E, compared with the case where it does not transmit continuously, after the virtual viewpoint video generation instruction 107 is given, the CPU 58 can acquire the reference imaging device information 46E in a short time and generate the virtual viewpoint video 46C. Also, since the CPU 58 generates the virtual viewpoint video 46C on the condition that the virtual viewpoint video generation instruction 107 is given, compared with the case of always generating the virtual viewpoint video 46C, the communication load and power consumption required for generating the virtual viewpoint video 46C can be reduced.

[0129] Also, each time the CPU 58 acquires the reference imaging device information 46E, it updates the reference imaging device information 46E. Therefore, compared with the case where the reference imaging device information 46E is not updated, the CPU 58 can switch the reference imaging device information 46E in response to the input of the reference imaging device identifier 35.

[0130] Also, the CPU 58 acquires the image quality information 46F indicating the image quality of the reference video 46D obtained by imaging the soccer field 24 by the reference imaging device 17. Further, the CPU 58 determines the image quality of the virtual viewpoint video 46C based on the image quality information 46F on the condition that the virtual viewpoint video generation instruction 107 is given. Therefore, the CPU 58 generates the virtual viewpoint video 46C having the image quality based on the image quality information 46F. According to this configuration, compared with the case where the image quality of the virtual viewpoint video 46C is not determined based on the image quality information 46F of the reference video 46D, the difference in image quality between the reference video 46D and the virtual viewpoint video 46C can be reduced.

[0131] In addition, the CPU 58 outputs a superimposed video 46G, which is obtained by superimposing a virtual viewpoint video 46C on a reference video 46D obtained by imaging the soccer field 24 by the reference imaging device 17, to the smart device 14 on the condition that a virtual viewpoint video generation instruction 107 is given. Therefore, since the field of view of the generated virtual viewpoint image is the same as or approximate to the field of view of the reference image, the virtual viewpoint image is superimposed on the reference image without having a difference in the field of view. According to this configuration, when switching the display from the reference video 46D to the virtual viewpoint video 46C, by outputting the superimposed video 46G, it is possible to reduce the visual discomfort given to the viewer 28 as compared with the case where the superimposed video 46G is not output.

[0132] The CPU 58 gradually changes the superimposing ratio between the reference video 46D and the virtual viewpoint video 46C in the superimposed video 46G. Therefore, the display can be gradually changed from the reference video 46D to the virtual viewpoint video 46C. According to this configuration, when switching the display from the reference video 46D to the virtual viewpoint video 46C, it is possible to further reduce the visual discomfort given to the viewer 28 as compared with the case where the reference video 46D and the virtual viewpoint video 46C are always superimposed at a constant ratio.

[0133] The CPU 58 outputs the virtual viewpoint video 46C to the display 78 of the smart device 14 and receives a change signal 120 that continuously changes at least one of the viewpoint position 42, the line-of-sight direction 44, and the field angle 43 in the output virtual viewpoint video 46C. According to this configuration, since at least one of the viewpoint position 42, the line-of-sight direction 44, and the field angle 43 is continuously changed by the change signal 120, it is possible to enhance the sense of presence in the virtual viewpoint video 46C as compared with the case where the viewpoint position 42, the line-of-sight direction 44, and the field angle 43 are not continuously changed.

[0134] The reference imaging device 17 is an imaging device capable of changing at least one of the position, the imaging direction, and the field angle. Therefore, it is possible to increase the degree of freedom in imaging the reference video 46D as compared with the case where the position, the imaging direction, and the field angle of the reference imaging device 17 cannot be changed.

[0135] The CPU 58 acquires reference imaging device information 46E based on a reference video 46D obtained by imaging the soccer field 24 by the reference imaging device 17. Therefore, the reference imaging device information 46E can be acquired more easily than in the case where the reference imaging device information 46E is not acquired based on the reference video 46D.

[0136] The reference imaging device information 46E is the position, imaging direction, and angle of view of the reference imaging device 17. Therefore, the difference in the field of view between the reference video 46D and the virtual viewpoint video 46C can be reduced more than in the case where the reference imaging device information 46E is not the position, imaging direction, and angle of view of the reference imaging device 17.

[0137] The reference imaging device 17 is one of a plurality of imaging devices. Therefore, the degree of freedom in setting which imaging device is to be set as the reference imaging device 17 can be increased compared to the case where the reference imaging device 17 is not one of the plurality of imaging devices.

[0138] The reference imaging device 17 is switchable among a plurality of imaging devices. Therefore, the degree of freedom in design of setting which imaging device is to be set as the reference imaging device 17 can be increased compared to the case where the reference imaging device 17 is not switchable.

[0139] The CPU 58 outputs the reference video 46D obtained by imaging the soccer field 24 by the reference imaging device 17, and generates a virtual viewpoint video 46C corresponding to the reference video 46D based on the reference imaging device information 46E on the condition that a virtual viewpoint video generation instruction 107 is given. Therefore, the difference in the field of view between the reference video 46D and the virtual viewpoint video 46C can be reduced compared to the case where a virtual viewpoint video 46C not corresponding to the reference video 46D is generated. C is generated. According to this configuration, the difference in the field of view between the reference video 46D and the virtual viewpoint video 46C can be reduced compared to the case where a virtual viewpoint video 46C not corresponding to the reference video 46D is generated.

[0140] In the above-described embodiment, an example of a form in which a plurality of imaging devices continuously transmit imaging device information has been described, but the technology of the present disclosure is not limited to this. For example, the imaging device information may be continuously transmitted by a server (not shown). In this case, the server is an example of the "transmission device" according to the technology of the present disclosure. The server holds the imaging device information of a plurality of imaging devices. Then, the control unit 100 receives the imaging device information transmitted from the server and stores it in the memory 62.

[0141] Also, in the above-described embodiment, the plurality of imaging devices continuously transmitted the imaging device information, but the plurality of imaging devices may transmit the imaging device information to the control unit 100 in response to a transmission request from the control unit 100. That is, the control unit 100 may issue a transmission request for the imaging device information only to the reference imaging device 17, and in response to the transmission request, the reference imaging device 17 may transmit its own imaging device information as the reference imaging device information 46E.

[0142] Also, in the above-described embodiment, the imaging device information of the plurality of imaging devices is stored in the memory 62, but the control unit 100 may store only the imaging device information of the reference imaging device 17 as the reference imaging device information 46E in the memory 62 or the server. In this case, since the reference imaging device 17 changes according to the operation of the video producer, each time the control unit 100 acquires the reference imaging device identifier 35, that is, each time the reference imaging device information 46E is acquired, the reference imaging device information 46E held in the memory 62 or the server is updated. The control unit 100 acquires the reference imaging device information 46E held in the server or the memory 62.

[0143] Also, in the above embodiment, the control unit 100 obtains the reference imaging device information 46E from the imaging device information transmitted by the plurality of imaging devices. However, the control unit 100 may obtain the reference imaging device information 46E based on the reference video 46D. In this case, for example, on the soccer field 24, signs indicating the distance from the reference position are provided at predetermined distances. On the soccer field 24, for example, with the center of the soccer field 24 as the reference position (0, 0), signs indicating the longitudinal distance X and the lateral distance Y of the soccer field 24 in the form of (X, Y) are provided in a grid pattern at regular intervals. Also, on each of the four walls surrounding the soccer field 24, signs indicating the horizontal distance X and the vertical distance Y from the reference position in the form of (X, Y) with the center in the longitudinal direction of the wall and the ground as the reference position (0, 0) are provided in a grid pattern at regular intervals. The control unit 100 can identify the field of view of the reference imaging device 17 by detecting the signs from the reference video 46D, and geometrically obtain the reference imaging device information 46E based on the identified field of view.

[0144] Note that in the above embodiment, an example of the form in which the reference video 46D is generated by the information processing device 12 has been described, but the technology of the present disclosure is not limited to this. The reference video 46D may be generated by a device different from the information processing device 12 (hereinafter referred to as a "video generation device").

[0145] Note that as an example of the "display" according to the technology of the present disclosure, the display 78 of the smart device 14 is cited. However, instead of the smart device 14, various devices with a display such as a head-up display, a head-mounted display, a personal computer, and / or a wearable terminal can also be adopted as the "display" according to the technology of the present disclosure.

[0146] In addition, in the above embodiment, the soccer stadium 22 was exemplified, but this is merely an example. Any location is acceptable as long as a plurality of imaging devices and a plurality of sound collection devices can be installed, such as a baseball field, a rugby field, a curling rink, an athletics stadium, a swimming pool, a concert hall, an outdoor music venue, and a theater venue.

[0147] In addition, in the above embodiment, the wireless communication method using the base station 20 was exemplified, but this is merely an example. The technology of the present disclosure is also applicable to a wired communication method using a cable.

[0148] In addition, in the above embodiment, the unmanned aerial vehicle 27 was exemplified, but the technology of the present disclosure is not limited thereto. For example, an imaging area may be imaged by an imaging device 18 suspended by a wire (for example, a self-propelled imaging device movable along a wire).

[0149] In addition, in the above embodiment, the computers 50 and 70 were exemplified, but the technology of the present disclosure is not limited thereto. For example, instead of the computers 50 and / or 70, a device including an ASIC, an FPGA, and / or a PLD may be applied. Further, instead of the computers 50 and / or 70, a combination of a hardware configuration and a software configuration may be used.

[0150] In addition, in the above embodiment, the information processing device side program is stored in the storage 60, but the technology of the present disclosure is not limited thereto. As an example, as shown in FIG. 20, the information processing device side program may be stored in an arbitrary portable storage medium 200 such as an SSD or a USB memory. In this case, the information processing device side program stored in the storage medium 200 is installed in the computer 50, and the CPU 58 executes the information processing device side processing according to the information processing device side program.

[0151] Alternatively, the information processing apparatus side program may be stored in a storage unit of another computer or a server device connected to the computer 50 via a communication network (not shown), and the information processing apparatus side program may be downloaded to the information processing apparatus 12 in response to a request from the information processing apparatus 12. In this case, the information processing apparatus side processing based on the downloaded information processing apparatus side program is executed by the CPU 58 of the computer 50.

[0152] In the above embodiment, the CPU 58 is exemplified, but the technology of the present disclosure is not limited thereto, and a GPU may be adopted. Further, instead of the CPU 58, a plurality of CPUs or a combination of a CPU and a GPU may be adopted. That is, the information processing apparatus side processing may be executed by one processor or a plurality of physically separated processors. Further, instead of the CPU 88, a GPU may be adopted, or a plurality of CPUs or a combination of a CPU and a GPU may be adopted, and various processes may be executed by one processor or a plurality of physically separated processors.

[0153] As the hardware resources for executing the information processing apparatus side processing, the following various processors can be used. As the processor, for example, as described above, a general-purpose processor that functions as a hardware resource for executing the information processing apparatus side processing according to software, that is, a program, such as a CPU, can be mentioned. Further, as other processors, for example, a dedicated electric circuit that is a processor having a circuit configuration dedicated to executing a specific process, such as an FPGA, a PLD, or an ASIC, can be mentioned. A memory is built in or connected to any of the processors, and any of the processors executes the information processing apparatus side processing by using the memory.

[0154] The hardware resources for executing the information processing apparatus side processing are among these various processors It may be composed of one of them, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of a plurality of FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the information processing device-side processing may be one processor.

[0155] As an example of configuring with one processor, first, as represented by computers such as clients and servers, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource for executing the information processing device-side processing. Second, as represented by an SoC or the like, there is a form in which a processor that realizes the functions of the entire system including a plurality of hardware resources for executing the information processing device-side processing is used. Thus, the information processing device-side processing is realized as a hardware resource using one or more of the above various processors.

[0156] Furthermore, as a more specific hardware structure of these various processors, an electric circuit combining circuit elements such as semiconductor elements can be used.

[0157] Also, the processing on the information processing device 12 side described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope of not departing from the gist.

[0158] The description and illustration shown above are detailed explanations of the part related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the explanations regarding the above-described configuration, function, operation, and effects are explanations regarding an example of the configuration, function, operation, and effects of the part related to the technology of the present disclosure. Therefore, it goes without saying that within the scope not departing from the gist of the technology of the present disclosure, the description and illustration shown above may be modified by deleting unnecessary parts, adding new elements, or making replacements. Also, in order to avoid complication and facilitate the understanding of the part related to the technology of the present disclosure, in the description and illustration shown above, explanations regarding common technical knowledge and the like that do not particularly require explanation for implementing the technology of the present disclosure are omitted.

[0159] In this specification, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0160] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually stated to be incorporated by reference.

Claims

1. A processor is provided. The processor, displaying a reference video sequence obtained by capturing an image of the imaging region by the reference imaging device on a display of the terminal; receiving an instruction to start generating a virtual viewpoint image while the reference video image is being displayed; acquiring information on a position, an imaging direction, and an angle of view of the reference imaging device at a time when the frame included in the reference moving image being displayed on the display was obtained at the timing when the instruction was received; The virtual viewpoint image having the same viewpoint position, line of sight direction, and angle of view as the position, image capturing direction, and angle of view of the reference image capturing device is displayed on the display based on a plurality of images obtained by capturing images of the image capturing area by a plurality of image capturing devices. Information processing device.

2. The processor, acquiring image quality information indicating an image quality of a first reference image obtained by imaging the imaging area by the reference imaging device; determining an image quality of the virtual viewpoint image based on the image quality information on the condition that the instruction has been given; The information processing device according to claim 1 .

3. The processor, on condition that the instruction is given, displays on the display a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging area by the reference imaging device.

3. The information processing device according to claim 1 or 2.

4. The processor gradually changes a ratio of the second reference image to the virtual viewpoint image in the superimposed image. The information processing device according to claim 3 .

5. displaying the virtual viewpoint image is realized by outputting the virtual viewpoint image to the display; The processor receives a change signal for continuously changing at least one of the viewpoint position, the line of sight direction, and the angle of view in the virtual viewpoint image output to the display. The information processing device according to any one of claims 1 to 4.

6. The reference imaging device is an imaging device that is capable of changing at least one of the position, the imaging direction, and the angle of view. The information processing device according to any one of claims 1 to 5.

7. The reference imaging device is one of the plurality of imaging devices. The information processing device according to any one of claims 1 to 6.

8. The reference imaging device is switchable among the plurality of imaging devices. The information processing device according to claim 7.

9. A method for operating an information processing device including a processor and a memory connected to or built into the processor, comprising: displaying a reference moving image obtained by capturing an image of the imaging region by the reference imaging device on a display of the terminal; receiving an instruction to start generating a virtual viewpoint image while the reference video image is being displayed; acquiring information regarding a position, an imaging direction, and an angle of view of the reference imaging device at a time when a frame included in the reference moving image being displayed on the display was obtained at a timing when the instruction is received; and and displaying the virtual viewpoint image, which has the same viewpoint position, line of sight direction, and angle of view as the position, image capturing direction, and angle of view as the reference image capturing device, on the display based on a plurality of images obtained by capturing images of the image capturing area by a plurality of image capturing devices. A method for operating an information processing device.

10. A computer that is applied to an information processing device including a processor and a memory connected to or built into the processor, displaying a reference moving image obtained by capturing an image of the imaging region by the reference imaging device on a display of the terminal; receiving an instruction to start generating a virtual viewpoint image while the reference video image is being displayed; acquiring information regarding a position, an imaging direction, and an angle of view of the reference imaging device at a time when a frame included in the reference moving image being displayed on the display was obtained at a timing when the instruction is received; and A program for executing a process including displaying, on the display, the virtual viewpoint image having the same viewpoint position, line of sight direction, and angle of view as the position, imaging direction, and angle of view of the reference imaging device based on multiple images obtained by imaging the imaging area by multiple imaging devices.

Images