Information processing device, method of operating the information processing device, and program

Abstract

To reduce the difference in the field of view between an image captured by a reference imaging device and a virtual viewpoint image.SOLUTION: An information processing device includes a processor and a memory connected to or built into the processor. The processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and generates a virtual viewpoint image on the basis of the reference imaging device information on condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging area by the plurality of imaging devices is given.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing apparatus, a method of operating the information processing apparatus, and a program.

Background Art

[0002] International Publication WO2018 / 021067 discloses an image processing apparatus including a virtual viewpoint image generation unit that generates a virtual viewpoint image captured at a virtual viewpoint composed of a virtual viewpoint position and a line-of-sight direction from a plurality of reference images captured at a plurality of viewpoint positions and line-of-sight directions.

[0003] Japanese Patent Application Laid-Open No. 2019-036790 discloses an image generation apparatus including: an image acquisition unit that acquires a group of images of a subject captured from a plurality of different positions; a viewpoint acquisition unit that acquires position information of a virtual viewpoint; a setting unit that sets the resolution of subject shape data according to the distance between a reference point serving as a reference for resolution setting and the virtual viewpoint; and a generation unit that generates a virtual viewpoint image of the subject from the virtual viewpoint using the subject shape data according to the resolution and the group of images of the subject.

Summary of the Invention

[0004] One embodiment of the technology of the present disclosure provides an information processing apparatus, a method of operating the information processing apparatus, and a program that can reduce the difference in the field of view between an image captured by a reference imaging device and a virtual viewpoint image.

Means for Solving the Problems

[0005] A first aspect of the technology of the present disclosure is an information processing apparatus including a processor and a memory connected to or incorporated in the processor, wherein the processor acquires reference imaging device information corresponding to the position, imaging direction, and angle of view of a reference imaging device, and generates a virtual viewpoint image based on the reference imaging device information on condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region by a plurality of imaging devices is given.

[0006] A second aspect of the technology of this disclosure is an information processing device according to the first aspect, wherein, when reference imaging device information is continuously transmitted by a transmitting device, a processor generates a virtual viewpoint image based on the reference imaging device information transmitted from the transmitting device, provided that an instruction has been given.

[0007] A third aspect of the technology of this disclosure is an information processing device according to the first or second aspect, wherein the processor updates the reference imaging device information each time it acquires the reference imaging device information.

[0008] A fourth aspect of the technology of this disclosure is an information processing device according to any one of the first to third aspects, wherein the processor acquires image quality information indicating the image quality of a first reference image obtained by imaging an imaging area with a reference imaging device, and determines the image quality of a virtual viewpoint image based on the image quality information, provided that an instruction is given.

[0009] A fifth aspect of the technology of this disclosure is an information processing device according to any one of the first to fourth aspects, wherein the processor outputs a superimposed image obtained by superimposing a virtual viewpoint image on a second reference image obtained by imaging an imaging area with a reference imaging device, provided that an instruction is given.

[0010] A sixth aspect of the technology of this disclosure is a processor that, in the superimposed image, and a second reference image This is an information processing device according to a fifth embodiment, which gradually changes the ratio with respect to the virtual viewpoint image.

[0011] A seventh aspect of the technology of this disclosure is an information processing device according to any one of the first to sixth aspects, wherein the processor outputs a virtual viewpoint image to a display and receives a change signal that continuously changes at least one of the viewpoint position, line of sight direction, and field of view in the output virtual viewpoint image.

[0012] An eighth aspect of the technology of this disclosure is an information processing device according to any one of the first to seventh aspects, wherein the reference imaging device is an imaging device in which at least one of the position, imaging direction, and field of view can be changed.

[0013] A ninth aspect of the technology of this disclosure is an information processing device according to any one of the first to eighth aspects, wherein the processor acquires reference imaging device information based on a third reference image obtained when an imaging area is imaged by a reference imaging device.

[0014] A tenth aspect of the technology of this disclosure is an information processing device relating to any one of the first to ninth aspects, wherein the reference imaging device information is the position, imaging direction, and field of view of the reference imaging device.

[0015] An eleventh aspect of the technology of this disclosure is an information processing device according to any one of the first to tenth aspects, wherein the reference imaging device is one of a plurality of imaging devices.

[0016] A twelfth aspect of the technology of this disclosure is an information processing device according to the eleventh aspect, wherein the reference imaging device is switchable between a plurality of imaging devices.

[0017] A thirteenth aspect of the technology of this disclosure is an information processing device according to any one of the first to twelfth aspects, wherein the processor outputs a fourth reference image obtained by imaging an imaging area with a reference imaging device, acquires reference imaging device information, and, subject to instructions, generates a virtual viewpoint image corresponding to the fourth reference image with reference to the reference imaging device information.

[0018] A fourteenth aspect of the technology of this disclosure is a method for operating an information processing device comprising a processor and a memory connected to or built into the processor, the method for operating an information processing device comprising acquiring reference imaging device information corresponding to the position, imaging direction, and field of view of a reference imaging device, and generating a virtual viewpoint image based on the reference imaging device information, provided that an instruction is given to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging area by a plurality of imaging devices.

[0019] A fifteenth aspect of the technology of this disclosure is a program applied to an information processing device comprising a processor and memory connected to or built into the processor, which causes a computer to perform a process that includes acquiring reference imaging device information corresponding to the position, imaging direction, and field of view of a reference imaging device, and, on the condition that an instruction is given to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging area by a plurality of imaging devices, generating a virtual viewpoint image based on the reference imaging device information. [Brief explanation of the drawing]

[0020] [Figure 1] This is a schematic perspective view showing an example of the external configuration of an information processing system. [Figure 2] This is a conceptual diagram illustrating an example of the relationship between information processing equipment, smart devices, receivers, unmanned aerial vehicles, and imaging devices included in an information processing system. [Figure 3] This block diagram shows an example of the electrical hardware configuration of an information processing device, and an example of the relationship between the information processing device and peripheral devices. [Figure 4] This block diagram shows an example of the electrical hardware configuration of a smart device. [Figure 5] This is a block diagram showing an example of the essential functions of an information processing device. [Figure 6] This is a block diagram illustrating an example of the process for generating a reference image. [Figure 7] This is a block diagram illustrating an example of the process for generating virtual viewpoint video. [Figure 8] It is a conceptual diagram for explaining an example of a display change that occurs when the display of a smart device is switched from a reference video to a virtual viewpoint video. [Figure 9] It is a conceptual diagram for explaining an example of a display change that occurs after the display of a smart device is switched to a virtual viewpoint video. [Figure 10] It is a conceptual diagram for explaining an example of the content of a change instruction issued by a user during the display of a virtual viewpoint video and the content of a change process performed in response to the change instruction. [Figure 11] It is a conceptual diagram for explaining an example of an instruction method for a viewpoint position change instruction and a viewpoint position change process. [Figure 12] It is a conceptual diagram for explaining an example of a camera path of a virtual imaging device in a viewpoint position change process. [Figure 13] It is a conceptual diagram for explaining an example of an instruction method for a viewpoint position change instruction and a line-of-sight direction change instruction, and a viewpoint position change process and a line-of-sight direction change process. [Figure 14] It is a conceptual diagram for explaining an example of a camera path of a virtual imaging device in a viewpoint position change process and a line-of-sight direction change process. [Figure 15] It is a conceptual diagram for explaining an example of an instruction method for an angle-of-view change instruction and an angle-of-view change process. [Figure 16] It is a conceptual diagram for explaining an example of an instruction method for an angle-of-view change instruction and another example of an angle-of-view change process. [Figure 17] It is a flowchart showing an example of the flow of a reference video generation process performed by a reference video generation unit. [Figure 18] It is a flowchart showing an example of the flow of a virtual viewpoint video generation process performed by a virtual viewpoint video generation unit. [Figure 19] It is a flowchart showing an example of the flow of a video distribution process performed by a control unit. [Figure 20] It is a block diagram showing an example of a mode in which an information processing device side program is installed from a storage medium storing the information processing device side program into a computer of the information processing device. [Modes for carrying out the invention]

[0021] An example of an embodiment relating to the technology of this disclosure will be described with reference to the attached drawings.

[0022] First, let's explain the terminology used in the following explanation.

[0023] CPU stands for "Central Processing Unit." RAM stands for "Random Access Memory." DRAM stands for "Dynamic Random Access Memory." SRAM stands for "Static Random Access Memory." ROM stands for "Read Only Memory." SSD stands for "Solid State Drive." HDD stands for "Hard Disk Drive." EEPROM stands for "Electrically Erasable and Programmable Read Only Memory." I / F stands for "Interface." The following are abbreviations. IC stands for "Integrated Circuit". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complement CCD stands for "Charge Coupled Device". EL stands for "Electro-Luminescence". GPU stands for "Graphics Processing Unit". LAN stands for "Local Area Network". 3D stands for "3 Dimension". HMD stands for "Head Mounted Display". USB stands for "Universal Serial Bus". fps is , refers to an abbreviation for "frames per second". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)." TDM is an abbreviation for "Time-Division Multiplexing."

[0024] In this specification, the meaning of “identical” includes not only a perfect match but also an approximate match that includes allowable design and manufacturing tolerances. In this specification, the meaning of “same” includes not only a perfect match but also an approximate match that includes allowable design and manufacturing tolerances. In this specification, the meaning of “linear” includes not only a perfect straight line but also an approximate straight line that includes allowable design and manufacturing tolerances. Furthermore, in this specification, the meaning of “curvilinear” includes not only a perfect curve but also an approximate curve that includes allowable design and manufacturing tolerances.

[0025] As an example, as shown in Figure 1, the information processing system 10 includes an information processing device 12, a smart device 14, multiple imaging devices 16, an imaging device 18, a wireless communication base station (hereinafter simply referred to as "base station") 20, and a receiver 34. Here, "smart device 14" refers to a portable multi-functional terminal such as a smartphone tablet, smartwatch (watch-type multi-functional terminal), and HMD-type multi-functional terminal. Here, the information processing device 12 is an example of an "information processing device" related to the technology of this disclosure. The multiple imaging devices 16 and imaging device 18 are examples of "multiple imaging devices" related to the technology of this disclosure. Here, the receiver 34 is given as an example, but the technology of this disclosure is not limited to this, and may be an electronic device with a display (for example, a smart device). Also, there may be more than one base station 20. Furthermore, the communication standards used at base stations include wireless communication standards such as LTE, wireless communication standards such as WiFi (802.11) and / or Bluetooth®, and wired communication standards such as TDM and / or Ethernet®.

[0026] The imaging devices 16 and 18 are imaging devices having a CMOS image sensor and are equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as CCD image sensors, may be used instead of the CMOS image sensor. Hereinafter, for the sake of explanation, when it is not necessary to distinguish between the imaging device 18 and the multiple imaging devices 16, they will be referred to as "multiple imaging devices" without reference numerals.

[0027] Multiple imaging devices 16 are installed within the soccer field 22. Each of the multiple imaging devices 16 is positioned to surround the soccer field 24 and images the area including the soccer field 24 as its imaging area. Here, the area including the soccer field 24 is an example of the "imaging area" related to the technology of this disclosure. Here, an example of a configuration in which each of the multiple imaging devices 16 is positioned to surround the soccer field 24 is given, but the technology of this disclosure is not limited to this, and the arrangement of the multiple imaging devices 16 is determined according to the virtual viewpoint image that is required to be generated by the viewer 28, etc. Multiple imaging devices 16 may be arranged to surround the entire soccer field 24, or multiple imaging devices 16 may be arranged to surround a specific part of it. The imaging device 18 is installed on an unmanned aerial vehicle (e.g., a multi-rotor-wing unmanned aerial vehicle) and images the area including the soccer field 24 from an aerial view as its imaging area. The imaging area, when viewed from above, including the soccer field 24, refers to the imaging surface of the imaging device 18 relative to the soccer field 24.

[0028] The information processing device 12 is installed in the control room 32. The multiple imaging devices 16 and the information processing device 12 are connected via a LAN cable 30. The information processing device 12 controls the multiple imaging devices 16 and acquires images obtained by each of the multiple imaging devices 16. Although this example illustrates a connection using a wired communication method via a LAN cable 30, the system is not limited to this and may use a wireless communication method.

[0029] The soccer stadium 22 has spectator seats 26 surrounding the soccer field 24, where viewers 28 are seated. The viewers 28 possess smart devices 14, which are used by the viewers 28. Although this description uses an example where the viewers 28 are located inside the soccer stadium 22, the technology of this disclosure is not limited to this, and the viewers 28 may be located outside the soccer stadium 22.

[0030] The base station 20 transmits and receives various types of information via radio waves with the information processing device 12 and the unmanned aerial vehicle 27. In other words, the information processing device 12 is wirelessly connected to the unmanned aerial vehicle 27 via the base station 20. By wirelessly communicating with the unmanned aerial vehicle 27 via the base station 20, the information processing device 12 controls the unmanned aerial vehicle 27 and acquires images obtained from the unmanned aerial vehicle 27 through imaging by the imaging device 18.

[0031] The base station 20 transmits various information to the receiver 34 via wireless communication. The information processing device 12 transmits various images to the receiver 34 via the base station 20, and the receiver 34 receives the various images transmitted from the information processing device 12 and displays the received images on the screen 34A. The receiver 34 is used, for example, for viewing by an unspecified number of spectators. The location of the receiver 34 may be inside the soccer stadium 22 or outside the soccer stadium 22 (for example, at a public viewing venue). Although this example describes a form in which various information is transmitted to the receiver 34 via wireless communication, the technology of this disclosure is not limited to this, and for example, various information may be transmitted to the receiver 34 via wired communication.

[0032] The information processing device 12 is a device equivalent to a server, and the smart device 14 is a device equivalent to a client terminal for the information processing device 12. The information processing device 12 and the smart device 14 communicate wirelessly with each other via the base station 20, so that the smart device 14 requests the information processing device 12 to provide various services, and the information processing device 12 provides the services to the smart device 14 in response to the requests from the smart device 14.

[0033] As an example, as shown in Figure 2, the information processing device 12 acquires an overhead view image 46A from the unmanned aerial vehicle 27 showing the area including the soccer field 24 as observed from above. The overhead view image 46A is a moving image obtained when the area including the soccer field 24 is captured by the imaging device 18 of the unmanned aerial vehicle 27 as an imaging area (hereinafter also simply referred to as the "imaging area") as an overhead view from above. Here, the case where the overhead view image 46A is a moving image is given as an example, but the overhead view image 46A is not limited to this and may be a still image showing the area including the soccer field 24 as observed from above.

[0034] The information processing device 12 acquires an image 46B from each of the multiple imaging devices 16, which shows the imaging area as observed from each of the multiple imaging devices 16. The image 46B is a moving image obtained by imaging the imaging area from each of the multiple imaging devices 16. Although the example given here shows the image 46B as a moving image, the image 46B is not limited to this, and may also be a still image showing the imaging area as observed from each of the multiple imaging devices 16. Images may also be used. Here, the overhead view 46A and the captured image 46B are examples of "multiple images" relating to the technology of this disclosure. Hereafter, for the sake of convenience of explanation, when it is not necessary to distinguish between the overhead view 46A and the multiple captured images 46B, they will be referred to as "multiple original images" without any reference numerals.

[0035] The information processing device 12 generates a virtual viewpoint image 46C based on multiple source images. The virtual viewpoint image 46C is an image showing the imaging area when the imaging area is observed from a specific viewpoint position and a specific line of sight direction. In the example shown in Figure 2, the virtual viewpoint image 46C refers to a virtual viewpoint image showing the imaging area when the soccer field 24 is observed from a viewpoint position 42 and line of sight direction 44 within the spectator seats 26 with a field of view 43. An example of a virtual viewpoint image 46C is a moving image using 3D polygons. The viewpoint position 42, line of sight direction 44, and field of view 43 are not fixed. That is, the viewpoint position 42, line of sight direction 44, and field of view 43 change according to instructions from the viewer 28, etc. Here, an example of a form in which the viewpoint position 42, line of sight direction 44, and field of view 43 are not fixed is given and explained, but the technology of this disclosure is not limited thereto, and the viewpoint position 42, line of sight direction 44, and field of view 43 may be fixed.

[0036] The information processing device 12 generates a moving image using 3D polygons by synthesizing multiple images obtained by imaging the imaging area using multiple imaging devices (here, as an example, an overhead image constituting the overhead image 46A and multiple imaging images constituting the multiple imaging images 46B (hereinafter, for the sake of explanation, when it is not necessary to distinguish between the overhead image and the multiple imaging images, they will be referred to as "multiple original images" without using reference numerals)). Based on the generated moving image using 3D polygons, the information processing device 12 generates a virtual viewpoint image corresponding to the case when the imaging area is observed from an arbitrary position and an arbitrary direction. In this embodiment, the information processing device 12 generates a virtual viewpoint image 46C that shows the subject when the subject is observed at the viewpoint position 42, the line of sight direction 44, and the field of view 43. In other words, the virtual viewpoint image 46C refers to an image corresponding to the image obtained when a virtual imaging device (hereinafter also referred to as the "virtual imaging device") installed at the viewpoint position 42 takes an image with the line of sight direction 44 as the imaging direction and the field of view 43. Here, a moving image is used as an example of the virtual viewpoint image 46C, but it is not limited to this and may also be a still image. The viewpoint position 42 is an example of a "viewpoint position" related to the technology of this disclosure. The line of sight direction 44 is an example of a "line of sight direction" related to the technology of this disclosure, and the field of view 43 is an example of a "field of view" related to the technology of this disclosure. Furthermore, the virtual viewpoint image 46C is an example of a "virtual viewpoint image" related to the technology of this disclosure.

[0037] Furthermore, while this example shows a configuration in which an overhead image 46A obtained by imaging device 18 is also used in the generation, the technology of this disclosure is not limited to this. For example, the overhead image 46A may not be used in the generation of the virtual viewpoint image 46C, and only the multiple imaged images 46B obtained by imaging each of the multiple imaging devices 16 may be used in the generation of the virtual viewpoint image 46C. In other words, the virtual viewpoint image 46C may be generated only from images obtained by imaging devices 16, without using the image obtained from imaging device 18. It should be noted that using the image obtained from imaging device 18 (e.g., a drone) makes it possible to generate a virtual viewpoint image with higher accuracy.

[0038] The information processing device 12 selectively transmits the reference video 46D (see, for example, Figures 6 and 7) and the virtual viewpoint video 46C, described later, to the smart device 14 and the receiver 34 as distributed video. For the sake of explanation, in the following, when it is not necessary to distinguish between the reference video 46D and the virtual viewpoint video 46C, they will also be referred to as "distributed video."

[0039] As an example, as shown in Figure 3, the information processing device 12 includes a computer 50, a receiving device 52, a display 53, a first communication interface 54, and a second communication interface 56. The computer 50 includes a CPU 58, storage 60, and memory 62. The PU58, storage 60, and memory 62 are connected via a bus 64. In the example shown in Figure 3, for illustrative purposes, a single bus is shown as bus 64, but there may be multiple buses. Bus 64 may also include a serial bus, or a parallel bus composed of a data bus, address bus, and control bus, etc. Memory 62 may also be built into the CPU 58. CPU 58 is an example of a "processor" related to the technology of this disclosure, and storage 60 and memory 62 are examples of "memory" related to the technology of this disclosure.

[0040] The CPU 58 controls the entire information processing unit 12. The storage 60 stores various parameters and programs. The storage 60 is a non-volatile storage device. Here, flash memory is used as an example of storage 60, but it is not limited to this, and may also be EEPROM, HDD, or SSD. The memory 62 is a storage device. Various information is temporarily stored in the memory 62. The memory 62 is used as work memory by the CPU 58. Here, DRAM is used as an example of memory 62, but it is not limited to this, and may also be other types of storage devices.

[0041] The reception device 52 receives instructions from the user of the information processing device 12. Examples of the reception device 52 include a touch panel, hard keys, and a mouse. The reception device 52 is connected to a bus 64, etc., and the instructions received by the reception device 52 are acquired by the CPU 58.

[0042] The display 53 is connected to the bus 64 and displays various information under the control of the CPU 58. An example of the display 53 is a liquid crystal display. However, other types of displays, such as organic EL displays, may also be used as the display 53.

[0043] The first communication interface 54 is connected to the LAN cable 30. The first communication interface 54 is implemented, for example, by a device having an FPGA. The first communication interface 54 is connected to the bus 64 and controls the exchange of various information between the CPU 58 and the multiple imaging devices 16. For example, the first communication interface 54 controls the multiple imaging devices 16 according to the requests of the CPU 58. The first communication interface 54 also acquires the captured images 46B (see Figure 2) obtained by each of the multiple imaging devices 16 and outputs the acquired captured images 46B to the CPU 58. Here, the first communication interface 54 is exemplified as a wired communication interface, but it may also be a wireless communication interface such as a high-speed wireless LAN.

[0044] The second communication interface 56 is wirelessly connected to the base station 20. The second communication interface 56 is implemented, for example, by a device having an FPGA. The second communication interface 56 is connected to the bus 64. The second communication interface 56 controls the exchange of various information between the CPU 58 and the unmanned aerial vehicle 27 via wireless communication through the base station 20. The second communication interface 56 also controls the exchange of various information between the CPU 58 and the smart device 14 via wireless communication through the base station 20. Furthermore, the second communication interface 56 controls the transmission of various images by the CPU 58 to the receiver 34 via wireless communication through the base station 20. At least one of the first communication interface 54 and the second communication interface 56 can be configured with a fixed circuit instead of an FPGA. At least one of the first communication interface 54 and the second communication interface 56 may also be a circuit configured with an ASIC, FPGA, and / or PLD, etc.

[0045] As an example, as shown in Figure 4, the smart device 14 includes a computer 70, a gyro sensor 74, a reception device 76, a display 78, a microphone 80, a speaker 82, an imaging device 84, and a communication I / F 86. The computer 70 has a CPU 88, The system includes storage 90 and memory 92, and the CPU 88, storage 90, and memory 92 are connected via a bus 94. In the example shown in Figure 4, for illustrative purposes, a single bus 94 is shown, but bus 94 may consist of multiple buses. Bus 94 may be a serial bus, or a parallel bus including a data bus, address bus, and control bus, etc.

[0046] The CPU 88 controls the entire smart device 14. The storage 90 stores various parameters and programs. The storage 90 is a non-volatile memory. Here, flash memory is used as an example of storage 90. Flash memory is merely one example; as storage 90, for example, various non-volatile memories such as magnetoresistive memory and / or ferroelectric memory can be used instead of flash memory or in combination with flash memory. The non-volatile memory may also be EEPROM, HDD, and / or SSD. Memory 92 temporarily stores various information and is used as work memory by the CPU 88. An example of memory 92 is RAM, but it is not limited to this and may be other types of memory.

[0047] The gyro sensor 74 measures the angle around the yaw axis of the smart device 14 (hereinafter also referred to as the "yaw angle"), the angle around the roll axis of the smart device 14 (hereinafter also referred to as the "roll angle"), and the angle around the pitch axis of the smart device 14 (hereinafter also referred to as the "pitch angle"). The gyro sensor 74 is connected to the bus 94, and the angle information indicating the yaw angle, roll angle, and pitch angle measured by the gyro sensor 74 is acquired by the CPU 88 via the bus 94, etc.

[0048] The receiving device 76 receives instructions from the user of the smart device 14 (for example, the viewer 28). Examples of the receiving device 76 include a touch panel 76A and hard keys. The receiving device 76 is connected to the bus 94, and the instructions received by the receiving device 76 are acquired by the CPU 88.

[0049] The display 78 is connected to the bus 94 and displays various information under the control of the CPU 88. An example of the display 78 is a liquid crystal display. However, other types of displays, such as organic EL displays, may also be used as the display 78.

[0050] The smart device 14 is equipped with a touch panel display, which is realized by a touch panel 76A and a display 78. That is, the touch panel display is formed by superimposing the touch panel 76A onto the display area of ​​the display 78, or by integrating the touch panel function into the display 78 ("in-cell" type).

[0051] Microphone 80 converts the collected sound into an electrical signal. Microphone 80 is connected to bus 94. The electrical signal obtained by converting the sound collected by microphone 80 is acquired by CPU 88 via bus 94.

[0052] Speaker 82 converts electrical signals into sound. Speaker 82 is connected to bus 94. Speaker 82 receives electrical signals output from CPU 88 via bus 94, converts the received electrical signals into sound, and outputs the resulting sound to the outside of smart device 14.

[0053] The imaging device 84 acquires an image of the subject by imaging the subject. The imaging device 84 is connected to the bus 94. The image obtained when the subject is imaged by the imaging device 84 The captured image is acquired by CPU88 via bus 94.

[0054] The communication interface 86 is wirelessly connected to the base station 20. The communication interface 86 is implemented by a device consisting of, for example, a circuit (e.g., an ASIC, FPGA, and / or PLD). The communication interface 86 is connected to the bus 94. The communication interface 86 controls the exchange of various information between the CPU 88 and an external device via wireless communication through the base station 20. Here, "external device" refers to, for example, an information processing device 12.

[0055] As an example, as shown in Figure 5, in the information processing device 12, the storage 60 stores a reference video generation processing program 60A, a virtual viewpoint video generation processing program 60B, and a video distribution processing program 60C. The virtual viewpoint video generation processing program 60B is an example of a "program" related to the technology of this disclosure. In the following, when it is not necessary to distinguish between the reference video generation processing program 60A, the virtual viewpoint video generation processing program 60B, and the video distribution processing program 60C, they will be referred to as the "information processing device side program" without using reference numerals.

[0056] The CPU 58 reads the information processing device program from the storage 60 and executes the read information processing device program in the memory 62. The CPU 58 exchanges various types of information with the smart device 14, the imaging device 16, and the unmanned aerial vehicle 27 according to the information processing device program executed in the memory 62, and also transmits various types of images to the receiver 34.

[0057] The CPU 58 reads the reference video generation processing program 60A from the storage 60 and executes the read reference video generation processing program 60A on the memory 62. The CPU 58 operates as the control unit 100 and the reference video generation unit 102 according to the reference video generation processing program 60A executed on the memory 62. By operating as the control unit 100 and the reference video generation unit 102, the CPU 58 executes the reference video generation process described later (see Figure 17).

[0058] The CPU 58 reads the virtual viewpoint image generation processing program 60B from the storage 60 and executes the read virtual viewpoint image generation processing program 60B on the memory 62. The CPU 58 operates as the control unit 100 and the virtual viewpoint image generation unit 104 in accordance with the virtual viewpoint image generation processing program 60B executed on the memory 62, thereby executing the virtual viewpoint image generation process described later (see Figure 18).

[0059] The CPU 58 reads the video distribution processing program 60C from the storage 60 and executes the read video distribution processing program 60C in memory 62. The CPU 58 operates as the control unit 100 according to the video distribution processing program 60C executed in memory 62. By operating as the control unit 100, the CPU 58 executes the video distribution processing described later (see Figure 19).

[0060] An example of the process for generating a reference image will be explained with reference to Figure 6. Multiple source images obtained by capturing images with multiple imaging devices are stored in the memory 62 of the information processing device 12. Each of the multiple source images is a moving image obtained by capturing images at a predetermined frame rate (e.g., 30fps), and consists of multiple still images. The memory 62 is provided with an area for storing a predetermined number of still images (e.g., several thousand frames) for each of the multiple source images, and older still images are overwritten with newly acquired still images. As a result, multiple source images for a predetermined amount of time are stored in the memory 62.

[0061] In the reference image generation process, the information processing device 12 receives instructions from the video producer (not shown) via the receiving device 52, and outputs a reference imaging device identifier 35 via the receiving device 52 according to the received instructions. The reference imaging device identifier 35 is an identifier that identifies one of several imaging devices.

[0062] Specifically, with multiple source images displayed side-by-side on the display 53, a video producer selects one image from the multiple source images via a touch panel. Then, an identifier associated with the imaging device 16 or 18 that captured the selected image 46B or overhead image 46A is output from the receiving device 52 as the reference imaging device identifier 35. For the sake of explanation, the imaging device 16 used to capture the selected image 46B, or the imaging device 18 used to capture the selected overhead image 46A, will be referred to as the reference imaging device 17. In other words, the reference imaging device 17 is one of multiple imaging devices. Here, the reference imaging device 17 is an example of a "reference imaging device" related to the technology of this disclosure.

[0063] The position, imaging direction, and field of view of the reference imaging device 17 are changeable. Furthermore, the reference imaging device 17 can be switched between multiple imaging devices. While this example illustrates a case where the position, imaging direction, and field of view of the reference imaging device 17 are changeable, the technology of this disclosure is not limited to this, and one or two of the position, imaging direction, and field of view of the reference imaging device 17 may be changeable. Also, while this example illustrates a case where one of multiple imaging devices is designated as the reference imaging device 17, the technology is not limited to this, and an imaging device other than the multiple imaging devices may be designated as the reference imaging device 17. In this case, the imaging device designated as the reference imaging device 17 may be a movable video camera (for example, a camera for television broadcasting) whose position, imaging direction, and field of view are controlled by a cameraman.

[0064] In the example shown in Figure 6, one of the multiple imaging devices 16 is designated as the reference imaging device 17. The reference image generation unit 102 reads the image 46B obtained by the reference imaging device 17 imaging the soccer field 24 from the memory 62, and generates a reference image 46D by superimposing information indicating the match situation (e.g., score, player information, and / or remaining time of the match) onto the read image 46B. Here, "reference image 46D" is, for example, live broadcast footage. The reference image 46D is an example of the "first reference image," "second reference image," "third reference image," and "fourth reference image" related to the technology of this disclosure.

[0065] In this case, the reference image 46D is generated based on the captured image 46B obtained by the imaging device 16 designated by the reference imaging device identifier 35. However, the captured image 46B itself may be used as the reference image 46D. Alternatively, instead of imaging device 16, imaging device 18 may be designated as the reference imaging device 17. In this case, the reference image 46D is generated by the reference image generation unit 102 based on the overhead image 46A.

[0066] The reference video generation unit 102 stores the generated reference video 46D in the memory 62. Here, the reference video 46D is a moving image that can be displayed at a predetermined frame rate (for example, 30fps). The control unit 100 reads the reference video 46D from the memory and outputs it to the receiver 34 via the second communication I / F 56 and the base station 20. In addition, if the control unit 100 has not received the virtual viewpoint video generation instruction 107 (see Figures 7 and 8) described later, it also outputs the reference video 46D to the smart device 14. In this case, the reference video 46D is displayed on the display 78 (see Figure 8).

[0067] Next, an example of the virtual viewpoint video generation process will be explained with reference to Figure 7. The touch panel 76A of the smart device 14 receives a virtual viewpoint video generation instruction 107 (see Figure 8) from the viewer 28. The control unit 100 receives the instruction from the touch panel 76A. A virtual viewpoint video generation instruction 107 is obtained. The virtual viewpoint video generation instruction 107 is obtained, for example, by a viewer 28 who is viewing a reference video 46D on the display 78 of a smart device 14, by turning on the instruction button 106 (see Figure 8) displayed on the display 78 via the touch panel 76A. Note that the virtual viewpoint video generation instruction 107 is an example of an "instruction" related to the technology of this disclosure.

[0068] The virtual viewpoint image generation instruction 107 is an instruction to start generating a virtual viewpoint image 46C based on multiple source images obtained by imaging the soccer field 24 with multiple imaging devices. For example, the virtual viewpoint image generation instruction 107 is given by the viewer 28 to the smart device 14 when instructing the information processing device 12 to start generating the virtual viewpoint image 46C and to display the generated virtual viewpoint image 46C on the display 78.

[0069] The control unit 100 acquires reference imaging device information 46E. The reference imaging device information 46E consists of the position, imaging direction, and field of view of the reference imaging device 17. Here, the position, imaging direction, and field of view of the reference imaging device 17 are given as examples of reference imaging device information 46E, but the technology of this disclosure is not limited to these, and may also be a position close to the position of the reference imaging device 17, a direction close to the imaging direction of the reference imaging device 17, and a field of view close to the field of view of the reference imaging device 17, or any information corresponding to the position, imaging direction, and field of view of the reference imaging device 17.

[0070] Each of the multiple imaging devices continuously transmits imaging device information corresponding to its own position, imaging direction, and field of view on the soccer field 24. The control unit 100 receives the imaging device information transmitted by the multiple imaging devices and stores it in the memory 62. Furthermore, the control unit 100 acquires the imaging device information transmitted by a designated reference imaging device 17 from among the multiple imaging devices by reading it from the memory 62 as reference imaging device information 46E. The reference imaging device information 46E is an example of "reference imaging device information" related to the technology of this disclosure. The reference imaging device 17 is an example of a "transmitting device" related to the technology of this disclosure. Here, "continuously transmitting imaging device information" means that imaging device information is transmitted multiple times. The transmission interval of imaging device information may be constant or it may not be constant.

[0071] The control unit 100 activates the extraction unit 108. The extraction unit 108 reads the reference image 46D from the memory 62 and obtains image quality information 46F indicating the image quality of the reference image 46D. The image quality information 46F obtained by the extraction unit 108 is stored in the memory 62. The image quality information 46F includes, for example, the resolution, brightness, and contrast of the image. Note that the image quality information 46F is an example of "image quality information" related to the technology of this disclosure.

[0072] Furthermore, the control unit 100 activates the virtual viewpoint image generation unit 104 on the condition that a virtual viewpoint image generation instruction 107 is given. The control unit 100 reads the captured image 46B, the overhead image 46A, the image quality information 46F, and the reference imaging device information 46E from the memory 62 and outputs them to the virtual viewpoint image generation unit 104.

[0073] The virtual viewpoint image generation unit 104 generates a virtual viewpoint image 46C using 3D polygons, based on the reference imaging device information 46E, by synthesizing multiple original images that constitute multiple original images, provided that a virtual viewpoint image generation instruction 107 is given. The virtual viewpoint image 46C is an image corresponding to the reference image 46D. For example, the virtual viewpoint image 46C is generated by imaging the soccer field 24 from a viewpoint position 42, line of sight direction 44, and field of view 43 that are the same as the position, imaging direction, and field of view included in the reference imaging device information 46E. That is, the virtual imaging device is the same as the reference imaging device 17, and the viewpoint position 42, line of sight direction 44, and field of view 43 of the virtual viewpoint image 46C are the same as those of the reference imaging device 17. It matches the position, imaging direction, and field of view of 7.

[0074] Furthermore, the virtual viewpoint video generation unit 104 determines the image quality of the virtual viewpoint video 46C to be generated based on the image quality information 46F, provided that a virtual viewpoint video generation instruction 107 is given. For example, the virtual viewpoint video generation unit 104 matches the resolution, brightness, and contrast of the virtual viewpoint video 46C to the resolution, brightness, and contrast included in the image quality information 46F. Here, an example of a form in which the resolution, brightness, and contrast of the virtual viewpoint video 46C are matched to the resolution, brightness, and contrast included in the image quality information 46F is given for explanation, but the technology of this disclosure is not limited to this, and the resolution, brightness, and contrast of the virtual viewpoint video 46C may be made to be close to the resolution, brightness, and contrast included in the image quality information 46F. The virtual viewpoint video generation unit 104 stores the generated virtual viewpoint video 46C in the memory 62.

[0075] Next, we will describe how the virtual viewpoint image 46C is displayed on the display 78 of the smart device 14. As an example, as shown in Figure 8, while the reference image 46D is displayed on the display 78, an instruction button 106 is displayed at the top of the display 78. When the viewer 28 touches the instruction button 106 on the touch panel 76A, a virtual viewpoint image generation instruction 107 is acquired from the smart device 14 by the information processing device 12.

[0076] The control unit 100, upon receiving a virtual viewpoint image generation instruction 107, activates the virtual viewpoint image generation unit 104 to generate a virtual viewpoint image 46C, as described above. The control unit 100 reads the reference image 46D obtained by the reference imaging device 17 and the virtual viewpoint image 46C generated based on the reference imaging device information 46E from the memory 62. The control unit 100 generates a superimposed image 46G by superimposing the reference image 46D and the virtual viewpoint image 46C. Specifically, the superimposed image 46G is an image in which the virtual viewpoint image 46C is superimposed on the reference image 46D. The control unit 100 outputs the superimposed image 46G to the smart device 14 via the second communication I / F 56 and the base station 20. The superimposed image 46G is displayed on the display 78 of the smart device 14. The superimposed image 46G is an example of a "superimposed image" related to the technology of this disclosure.

[0077] Furthermore, the control unit 100 gradually changes the ratio of the reference image 46D and the virtual viewpoint image 46C in the superimposed image 46G. As an example, as shown in Figure 9, after receiving the virtual viewpoint image generation instruction 107, the control unit 100 generates a superimposed image by superimposing the reference image and the virtual viewpoint image in a ratio of, for example, 9:1, and outputs it to the smart device 14. Here, the ratio refers to, for example, the ratio of the average pixel value of the reference image in the superimposed image 46G to the average pixel value of the virtual viewpoint image in the superimposed image 46G. In other words, the ratio here refers to the ratio of the degree to which the reference image occupies and the degree to which the virtual viewpoint image occupies, on a pixel-by-pixel basis in the superimposed image 46G. Here, the reference image refers to the image of one frame that constitutes the reference image 46D. The virtual viewpoint image refers to the image of one frame that constitutes the virtual viewpoint image 46C. Note that the ratio is an example of the "proportion" related to the technology of this disclosure.

[0078] The superimposed image generated by the control unit 100 according to a 9:1 ratio is output to the smart device 14. For example, if the output frame rate of the superimposed video 46G and virtual viewpoint video 46C to the smart device 14 is 30fps, the control unit 100 outputs the superimposed image to the smart device 14 while changing the ratio in units of 3 frames. For example, first the control unit 100 outputs the superimposed image for 3 frames generated at a 9:1 ratio to the smart device 14.

[0079] Next, the control unit 100 generates a superimposed image for 3 frames by overlaying the reference image and the virtual viewpoint image in an 8:2 ratio, and outputs it to the smart device 14. Thereafter, the control unit 100 gradually changes the ratio of the reference image and the virtual viewpoint image to 7:3, 6:4, ..., 1:9. The generated superimposed images are output to the smart device 14 in sets of three frames each. Subsequently, the control unit 100 outputs the virtual viewpoint video 46C, which is composed of virtual viewpoint images, to the smart device 14.

[0080] For example, alpha blending is applied when superimposing a reference image and a virtual viewpoint image. Alpha blending is a technique that combines two images according to a coefficient (alpha value). The alpha value is a value in the range of 0 to 1, where an image with an alpha value of 1 is completely opaque, and an image with an alpha value of 0 is completely transparent. Therefore, if the reference image and the virtual viewpoint image are superimposed in a 9:1 ratio, for example, the alpha value of the reference image would be set to 0.9 and the alpha value of the virtual viewpoint image would be set to 0.1.

[0081] In the example shown in Figure 9, the smart device 14 depicted at the top of Figure 9 is shown at the moment the instruction button 106 is touched, and the display 78 shows a superimposed image generated by overlaying the reference image and the virtual viewpoint image in a 9:1 ratio. The smart device 14 depicted in the center of Figure 9 is shown at a time, for example, 0.5 seconds after touching the instruction button 106, and the display 78 shows a superimposed image generated by overlaying the reference image and the virtual viewpoint image in a 5:5 ratio. The smart device 14 depicted at the bottom of Figure 9 is shown at a time, for example, 1.0 second after touching the instruction button 106, and the display on the display 78 has switched from the superimposed image to the virtual viewpoint image. In this way, the display of the smart device 14 gradually changes from the reference image 46D to the superimposed image and then to the virtual viewpoint image 46C over a period of 1 second.

[0082] When the virtual viewpoint image 46C is displayed on the display 78, an instruction button 106 appears at the top of the display 78. The instruction button 106 is operated when the display on the display 78 is changed from the virtual viewpoint image 46C to the reference image 46D. When the viewer 28 touches the instruction button 106 on the touch panel 76A, a reference image generation instruction 109 is acquired by the information processing device 12 from the smart device 14. The control unit 100, upon receiving the reference image generation instruction 109, stops the generation and output of the virtual viewpoint image 46C and outputs the reference image 46D to the smart device 14.

[0083] As an example, as shown in Figure 10, the control unit 100 outputs a virtual viewpoint image 46C to the display 78 of the smart device 14 and receives a change signal 120 that continuously changes at least one of the viewpoint position 42, line of sight direction 44, and field of view 43 in the output virtual viewpoint image 46C. Here, the display 78 is an example of a "display" according to the technology of this disclosure. The change signal 120 is an example of a "change signal" according to the technology of this disclosure.

[0084] In the virtual viewpoint image 46C displayed on the display 78, change instructions 110 from the viewer 28 are received via the touch panel 76A. The change instructions 110 are broadly classified into three types: a viewpoint position change instruction 110A for changing the viewpoint position 42, a gaze direction change instruction 110B for changing the gaze direction 44, and a field of view change instruction 110C for changing the field of view 43. The change instructions 110 received via the touch panel 76A are output from the smart device 14 to the control unit 100.

[0085] The control unit 100 generates a change signal 120 that indicates a change instruction 110 input from the smart device 14. The change signal 120 is broadly classified into a viewpoint position change signal 120A indicating a viewpoint position change instruction 110A, a gaze direction change signal 120B indicating a gaze direction change instruction 110B, and a field of view change signal 120C indicating a field of view change instruction 110C. The change signal 120 is output from the control unit 100 to the virtual viewpoint image generation unit 104.

[0086] The virtual viewpoint image generation unit 104 receives a viewpoint position change signal 120A from the control unit 100. If this occurs, the virtual viewpoint image generation unit 104 executes a viewpoint position change process 104A to change the viewpoint position 42. The virtual viewpoint image generation unit 104 executes a viewpoint direction change process 104B to change the viewpoint direction 44 when it receives a viewpoint direction change signal 120B from the control unit 100. The virtual viewpoint image generation unit 104 executes a field of view change process 104C to change the field of view 43 when it receives a field of view change signal 120C from the control unit 100.

[0087] As an example, as shown in Figure 11, the viewpoint position change instruction 110A is an instruction to change the viewpoint position 42 of the virtual viewpoint image 46C. The viewpoint position change instruction 110A is received by the touch panel 76A when the viewer 28 slides their finger in a straight line on the touch panel 76A. For example, as shown in the smart device 14 depicted at the top of Figure 11, the viewpoint position change instruction 110A is given to the smart device 14 when the viewer 28 slides their finger on the touch panel 76A in the direction of the dotted arrow.

[0088] The virtual viewpoint image 46C displayed on the smart device 14 shown at the top of Figure 11 is an image captured by a virtual imaging device positioned at viewpoint position 42A. When moving the viewpoint position 42A of the virtual imaging device to viewpoint position 42B, the viewer 28 gives a viewpoint position change instruction 110A to the smart device 14 by performing a swipe motion on the touch panel 76A, sliding their finger continuously in the opposite direction to the direction of viewpoint position movement.

[0089] Thus, when a viewpoint position change instruction 110A is given to the smart device 14, the viewpoint position change process 104A is performed by the virtual viewpoint image generation unit 104, as described above. As a result of the viewpoint position change process 104A performed by the virtual viewpoint image generation unit 104, the virtual imaging device moves from viewpoint position 42A to viewpoint position 42B, as shown in Figure 12 as an example. The virtual viewpoint image 46C generated by the virtual viewpoint image generation unit 104 with reference to viewpoint position 42B is obtained by being virtually imaged by the virtual imaging device positioned at viewpoint position 42B.

[0090] As an example, as shown in Figure 13, the viewer 28 can also give a gaze direction change instruction 110B to the smart device 14 along with a gaze position change instruction 110A. The gaze direction change instruction 110B is an instruction to change the gaze direction 44 of the virtual viewpoint image 46C. To give the gaze direction change instruction 110B to the smart device 14, for example, the viewer 28 slides their finger in a curved motion on the touch panel 76A. As shown at the top of Figure 13, the viewer 28 slides their finger on the touch panel 76A in the direction of the dotted arrow, and the gaze position change instruction 110A is received by the touch panel 76A.

[0091] The virtual viewpoint image 46C displayed on the smart device 14 shown at the top of Figure 13 is an image obtained by capturing the line of sight direction 44C with a virtual imaging device positioned at viewpoint position 42C. When moving the viewpoint position 42C of the virtual imaging device to viewpoint position 42D and changing the line of sight direction 44C of the virtual imaging device to the line of sight direction 44D, the viewer 28 gives the smart device 14 a viewpoint position change instruction 110A and a line of sight direction change instruction 110B by swiping their finger continuously in the opposite direction to the direction of viewpoint position movement on the touch panel 76A.

[0092] Thus, when the smart device 14 is given a viewpoint position change instruction 110A and a gaze direction change instruction 110B, the virtual viewpoint image generation unit 104 performs viewpoint position change processing 104A and gaze direction change processing 104B. As a result, as shown in Figure 14 as an example, the viewpoint position of the virtual imaging device is changed from viewpoint position 42C to viewpoint position 42D, and the gaze direction of the virtual imaging device is changed from gaze direction 44C to gaze direction 44D. The virtual viewpoint image 46C generated by the virtual viewpoint image generation unit 104 based on viewpoint position 42D and gaze direction 44D is then processed by the virtual imaging device positioned at viewpoint position 42D. This is obtained by virtually imaging direction 44D.

[0093] As an example, as shown in Figures 15 and 16, the field of view change instruction 110C is an instruction to change the field of view 43 of the virtual viewpoint image 46C. In order to give the field of view change instruction 110C to the smart device 14, the viewer 28 performs a pinch-in action on the touch panel 76A, where they pinch the screen with two fingers, or a pinch-out action, where they pinch and release the screen. A pinch-out action is performed to narrow the field of view, and a pinch-in action is performed to widen the field of view.

[0094] In the example shown in Figure 15, the viewer 28 performs a pinch-out gesture on the touch panel 76A displaying the virtual viewpoint image 46C. This pinch-out gesture on the touch panel 76A sends a field-of-view change instruction 110C to the smart device 14, instructing it to narrow the field of view. As a result of this instruction, the virtual viewpoint image generation unit 104 performs a field-of-view change process 104C in response to the pinch-out gesture, as described above. This process results in the virtual viewpoint image generation unit 104 generating the virtual viewpoint image 46C so that the field of view narrows in accordance with the pinch-out gesture, as shown in Figure 15 as an example.

[0095] In the example shown in Figure 16, the viewer 28 performs a pinch-in gesture on the touch panel 76A displaying the virtual viewpoint image 46C. This pinch-in gesture on the touch panel 76A triggers a field-of-view change instruction 110C, which instructs the smart device 14 to widen the field of view. As a result of this instruction, the virtual viewpoint image generation unit 104 performs a field-of-view change process 104C in response to the pinch-in gesture. This process, performed by the virtual viewpoint image generation unit 104, generates the virtual viewpoint image 46C in such a way that the field of view widens in accordance with the pinch-in gesture, as shown in Figure 16 as an example.

[0096] Next, an example of the flow of the reference image generation process, virtual viewpoint image generation process, and video distribution process performed in the information processing device 12 will be explained with reference to Figures 17 to 19.

[0097] As an example, the reference video generation process shown in Figure 17 is executed by the CPU 58 according to the reference video generation process program 60A when an instruction to execute the reference video generation process is received by the receiving device 52.

[0098] In the reference video generation process shown in Figure 17, first, in step S10, the control unit 100 determines whether or not the image generation timing has arrived. An example of the image generation timing is a timing defined according to the frame rate. For example, if the frame rate of the information processing device 12 is 30fps, the image generation timing arrives every 1 / 30th of a second. In step S10, if the image generation timing has not arrived, the determination is denied, and the reference video generation process proceeds to step S15. In step S10, if the image generation timing has arrived, the determination is affirmed, and the reference video generation process proceeds to step S11.

[0099] In step S11, the control unit 100 obtains the reference imaging device identifier 35. The reference imaging device identifier 35 is an identifier that specifies one of several imaging devices, and is received irregularly, for example, by the operation of the reception device 52 by the video producer, and overwritten and saved in the memory 62. The control unit 100 obtains the reference imaging device identifier 35 received at the most recent timing from the memory 62. After that, the reference video generation process proceeds to step We will be moving to S12.

[0100] In step S12, the control unit 100 reads from the memory 62 an image or overhead image for one frame obtained by the imaging device identified by the reference imaging device identifier 35, i.e., the reference imaging device 17, and outputs it to the reference image generation unit 102. After that, the reference image generation process proceeds to step S13.

[0101] In step S13, the reference image generation unit 102 superimposes match situation information onto the captured image or overhead image obtained by the reference imaging device 17 to generate a reference image for one frame. The match situation information includes, for example, the score, player information, and / or the remaining time of the match. After that, the reference image generation process proceeds to step S14.

[0102] In step S14, the reference image generation unit 102 stores the reference image for one frame generated in step S13 in the memory 62. After that, the reference image generation process proceeds to step S15.

[0103] In step S15, the control unit 100 determines whether the conditions for terminating the reference video generation process (hereinafter referred to as "reference video generation process termination conditions") have been met. An example of a reference video generation process termination condition is that an instruction to terminate the generation of the reference video 46D has been received by the receiving device 52 (see Figure 3). If the reference video generation process termination conditions are not met in step S15, the determination is denied, and the reference video generation process proceeds to step S10. If the reference video generation process termination conditions are met in step S15, the determination is affirmed, and the reference video generation process ends.

[0104] As an example, the virtual viewpoint image generation process shown in Figure 18 is executed by the CPU 58 according to the virtual viewpoint image generation process program 60B when a virtual viewpoint image generation instruction 107 is received by the touch panel 76A of the smart device 14.

[0105] In the virtual viewpoint video generation process shown in Figure 18, first, in step S20, the control unit 100 determines whether or not the virtual viewpoint video generation instruction 107 has been received by the touch panel 76A of the smart device 14. If the virtual viewpoint video generation instruction 107 has not been received by the touch panel 76A of the smart device 14 in step S20, the determination is denied and the determination in step S20 is repeated. If the virtual viewpoint video generation instruction 107 has been received by the touch panel 76A of the smart device 14 in step S20, the determination is affirmed and the virtual viewpoint video generation process proceeds to step S21. Note that step S20 may be a later step than step S22, and specifically, it may be placed between steps S22 and S23.

[0106] In step S21, the control unit 100 reads the reference imaging device information 46E from the memory 62. The reference imaging device information 46E is information corresponding to the position, imaging direction, and field of view of the reference imaging device 17. After that, the virtual viewpoint image generation process proceeds to step S22.

[0107] In step S22, the control unit 100 determines the viewpoint position 42, line of sight direction 44, and field of view 43 of the virtual imaging device based on the reference imaging device information 46E. For example, the control unit 100 determines the viewpoint position 42, line of sight direction 44, and field of view 43 of the virtual imaging device so that the field of view of the image obtained by the virtual imaging device is the same as the field of view of the image obtained by the reference imaging device 17. After that, the reference image generation process proceeds to step S23.

[0108] In step S23, the control unit 100 determines whether or not the image generation timing has arrived. If the image generation timing has not arrived in step S23, the determination is rejected. Then, the virtual viewpoint image generation process moves to step S29. If the image generation timing arrives in step S23, the determination is affirmed, and the virtual viewpoint image generation process moves to step S24.

[0109] In step S24, the control unit 100 reads multiple source images from the memory 62 based on the viewpoint position 42, line of sight direction 44, and field of view 43 defined in step S22. That is, the control unit 100 reads multiple source images from the memory 62 that are necessary to generate a virtual viewpoint image 46C of the field of view defined by the viewpoint position 42, line of sight direction 44, and field of view 43, and outputs them to the virtual viewpoint image generation unit 104. After that, the virtual viewpoint image generation process proceeds to step S25.

[0110] In step S25, the virtual viewpoint image generation unit 104 generates a virtual viewpoint image for one frame of the field of view defined by the viewpoint position 42, line of sight direction 44, and field of view 43 determined in step S22, from a plurality of original images. That is, the virtual viewpoint image generation unit 104 generates a virtual viewpoint image having the same field of view as the reference image. After that, the virtual viewpoint image generation process proceeds to step S26.

[0111] In step S26, the virtual viewpoint image generation unit 104 stores the virtual viewpoint image for one frame generated in step S25 in the memory 62. After that, the virtual viewpoint image generation process proceeds to step S27.

[0112] In step S27, the control unit 100 determines whether or not the change instruction 110 has been received by the smart device 14. If the change instruction 110 has not been received by the smart device 14 in step S27, the determination is denied and the virtual viewpoint image generation process proceeds to step S29. If the change instruction 110 has been received by the smart device 14 in step S27, the determination is affirmed and the virtual viewpoint image generation process proceeds to step S28.

[0113] In step S28, the control unit 100 determines the viewpoint position 42, line of sight direction 44, and field of view 43 of the virtual imaging device based on the change instruction 110. That is, the viewpoint position 42, line of sight direction 44, and field of view 43 determined in step S22 are discarded, and the viewpoint position 42, line of sight direction 44, and field of view 43 determined in step S28 become effective. In other words, in step S25, a virtual viewpoint image for one frame of the field of view defined by the viewpoint position 42, line of sight direction 44, and field of view 43 determined in step S28 is generated. After the processing in step S28 is executed, the virtual viewpoint image generation process proceeds to step S29.

[0114] In step S29, the control unit 100 determines whether the conditions for terminating the virtual viewpoint video generation process (hereinafter referred to as "virtual viewpoint video generation process termination conditions") have been met. An example of a virtual viewpoint video generation process termination condition is that an instruction to terminate the generation of the virtual viewpoint video 46C has been received by the receiving device 52 (see Figure 3). If the virtual viewpoint video generation process termination conditions are not met in step S29, the determination is denied, and the virtual viewpoint video generation process proceeds to step S23. If the virtual viewpoint video generation process termination conditions are met in step S29, the determination is affirmed, and the virtual viewpoint video generation process ends.

[0115] As an example, the video distribution process shown in Figure 19 is executed by the CPU 58 according to the video distribution program 60C when an instruction to start the video distribution process is received by the touch panel 76A of the smart device 14.

[0116] In the video distribution process shown in Figure 19, first, in step S31, the control unit 100 outputs an image. The system determines whether or not the timing has arrived. An example of an image output timing is a timing divided into time intervals defined by the output frame rate. If the image output timing has not arrived in step S31, the determination is denied, and the determination in step S31 is repeated. If the image output timing has arrived in step S31, the determination is affirmed, and the video distribution process proceeds to step S32.

[0117] In step S32, the control unit 100 reads a reference image for one frame from the memory 62 and outputs it to the smart device 14. After that, the video distribution process proceeds to step S33.

[0118] In step S33, the control unit 100 determines whether the conditions for terminating the video distribution process (hereinafter referred to as "video distribution processing termination conditions") have been met. An example of a video distribution processing termination condition is that an instruction to terminate the video distribution process has been received by the receiving device 52 or 76. If the video distribution processing termination conditions are not met in step S33, the determination is denied, and the video distribution process proceeds to step S34. If the video distribution processing termination conditions are met in step S33, the determination is affirmed, and the video distribution process ends.

[0119] In step S34, the control unit 100 determines whether or not the virtual viewpoint video generation instruction 107 has been received by the touch panel 76A of the smart device 14. If the virtual viewpoint video generation instruction 107 has not been received by the touch panel 76A of the smart device 14 in step S34, the determination is denied and the video distribution process proceeds to step S31. If the virtual viewpoint video generation instruction 107 has been received by the touch panel 76A of the smart device 14 in step S34, the determination is affirmed and the video distribution process proceeds to step S35.

[0120] In step S35, the control unit 100 determines whether or not the image output timing has arrived. If the image output timing has not arrived in step S35, the determination is denied and the determination in step S35 is repeated. If the image output timing has arrived in step S35, the determination is affirmed and the video distribution process proceeds to step S36.

[0121] In step S36, the control unit 100 reads one frame of virtual viewpoint image from memory 62. After that, the video distribution process proceeds to step S37.

[0122] In step S37, the control unit 100 determines whether the next display frame is the initial display frame that occurred shortly after receiving the virtual viewpoint video generation instruction 107. For example, if the next frame is the Nth display frame (where N is a natural number greater than or equal to 1) and is less than a specific number of frames (e.g., 30) (N < 30), the determination is affirmed and the video distribution process proceeds to step S38. On the other hand, if the next display frame is the Nth display frame and is greater than or equal to a specific number of frames (N ≥ 30), the determination is denied and the video distribution process proceeds to step S39. Although the specific number of frames has been described as "30," the technology of this disclosure is not limited to this, and the specific number of frames may be set to any natural number.

[0123] In step S38, the control unit 100 generates a superimposed image by overlaying a virtual viewpoint image onto the reference image, and outputs the generated superimposed image to the smart device 14. After that, the video distribution process proceeds to step S40.

[0124] In step S39, the control unit 100 outputs the virtual viewpoint image for one frame read in step S36 to the smart device 14. The video distribution process then proceeds to step S40. Migrate.

[0125] In step S40, the control unit 100 determines whether the conditions for terminating the video distribution process (hereinafter referred to as "video distribution processing termination conditions") have been met. An example of a video distribution processing termination condition is that the video distribution application has been terminated. If the video distribution processing termination conditions are not met in step S40, the determination is denied, and the video distribution process proceeds to step S41. If the video distribution processing termination conditions are met in step S40, the determination is affirmed, and the video distribution process ends.

[0126] In step S41, the control unit 100 determines whether or not the reference video generation instruction 109 has been received by the touch panel 76A of the smart device 14. If the reference video generation instruction 109 has not been received by the touch panel 76A of the smart device 14 in step S41, the determination is denied and the video distribution process proceeds to step S35. If the reference video generation instruction 109 has been received by the touch panel 76A of the smart device 14 in step S41, the determination is affirmed and the video distribution process proceeds to step S31.

[0127] As described above, the information processing device 12 comprises a CPU 58 and a memory 62 connected to or built into the CPU 58. The CPU 58 acquires reference imaging device information 46E corresponding to the position, imaging direction, and field of view of the reference imaging device 17. Furthermore, the CPU 58 generates a virtual viewpoint image 46C based on the reference imaging device information 46E, provided that a virtual viewpoint image generation instruction 107 is given, which is an instruction to start generating a virtual viewpoint image 46C based on multiple original images. Therefore, a virtual viewpoint image 46C having the same or similar field of view as the reference imaging device information 46E is generated. With this configuration, the difference in the field of view between the reference image 46D and the virtual viewpoint image 46C can be reduced compared to the case where the virtual viewpoint image 46C is generated based on the position, imaging direction, and field of view of an imaging device 16 or 18 that is different from the reference imaging device 17 among the multiple imaging devices.

[0128] Furthermore, if the reference imaging device information 46E is continuously transmitted by the reference imaging device 17, the CPU 58 generates a virtual viewpoint image 46C based on the reference imaging device information 46E transmitted from the reference imaging device 17, provided that a virtual viewpoint image generation instruction 107 is given. Since the reference imaging device 17 continuously transmits the reference imaging device information 46E, the CPU 58 can acquire the reference imaging device information 46E and generate the virtual viewpoint image 46C in a shorter time after the virtual viewpoint image generation instruction 107 is given, compared to when it is not continuously transmitted. In addition, since the CPU 58 generates the virtual viewpoint image 46C only when the virtual viewpoint image generation instruction 107 is given, the communication load and power consumption required to generate the virtual viewpoint image 46C can be reduced compared to when the virtual viewpoint image 46C is always generated.

[0129] Furthermore, the CPU 58 updates the reference imaging device information 46E each time it acquires the reference imaging device information 46E. Therefore, the CPU 58 can switch the reference imaging device information 46E in response to the input of the reference imaging device identifier 35, compared to when the reference imaging device information 46E is not updated.

[0130] Furthermore, the CPU 58 acquires image quality information 46F indicating the image quality of the reference image 46D obtained by the reference imaging device 17 capturing the soccer field 24. Then, the CPU 58 determines the image quality of the virtual viewpoint image 46C based on the image quality information 46F, provided that a virtual viewpoint image generation instruction 107 is given. Therefore, the CPU 58 generates a virtual viewpoint image 46C having the image quality based on the image quality information 46F. This configuration reduces the difference in image quality between the reference image 46D and the virtual viewpoint image 46C compared to the case where the image quality of the virtual viewpoint image 46C is not determined based on the image quality information 46F of the reference image 46D.

[0131] Furthermore, the CPU 58, upon receiving a virtual viewpoint image generation instruction 107, outputs a superimposed image 46G to the smart device 14, which is created by superimposing the virtual viewpoint image 46C onto the reference image 46D obtained by imaging the soccer field 24 with the reference imaging device 17. Therefore, since the field of view of the generated virtual viewpoint image is the same as or similar to that of the reference image, the virtual viewpoint image is superimposed on the reference image without any difference in field of view. With this configuration, the output of the superimposed image 46G when switching the display from the reference image 46D to the virtual viewpoint image 46C reduces the visual discomfort given to the viewer 28 compared to when the superimposed image 46G is not output.

[0132] The CPU 58 gradually changes the superposition ratio of the reference image 46D and the virtual viewpoint image 46C in the superimposed image 46G. Therefore, the display can be gradually changed from the reference image 46D to the virtual viewpoint image 46C. With this configuration, when switching the display from the reference image 46D to the virtual viewpoint image 46C, the visual discomfort caused to the viewer 28 can be further reduced compared to when the reference image 46D and the virtual viewpoint image 46C are always superimposed at a constant ratio.

[0133] The CPU 58 outputs the virtual viewpoint image 46C to the display 78 of the smart device 14 and receives a change signal 120 that continuously changes at least one of the viewpoint position 42, gaze direction 44, and field of view 43 in the outputted virtual viewpoint image 46C. With this configuration, since at least one of the viewpoint position 42, gaze direction 44, and field of view 43 is continuously changed by the change signal 120, the sense of realism in the virtual viewpoint image 46C can be enhanced compared to the case where the viewpoint position 42, gaze direction 44, and field of view 43 are not continuously changed.

[0134] The reference imaging device 17 is an imaging device in which at least one of its position, imaging direction, and field of view can be changed. Therefore, the degree of freedom in imaging the reference image 46D can be increased compared to a case where the position, imaging direction, and field of view of the reference imaging device 17 cannot be changed.

[0135] The CPU 58 acquires reference imaging device information 46E based on the reference image 46D obtained by the soccer field 24 being imaged by the reference imaging device 17. Therefore, the reference imaging device information 46E can be acquired more easily than in the case where the reference imaging device information 46E is not acquired based on the reference image 46D.

[0136] The reference imaging device information 46E consists of the position, imaging direction, and field of view of the reference imaging device 17. Therefore, compared to the case where the reference imaging device information 46E does not consist of the position, imaging direction, and field of view of the reference imaging device 17, the difference in the field of view between the reference image 46D and the virtual viewpoint image 46C can be further reduced.

[0137] The reference imaging device 17 is one of several imaging devices. Therefore, compared to the case where the reference imaging device 17 is not one of several imaging devices, the degree of freedom in setting which imaging device to designate as the reference imaging device 17 is increased.

[0138] The reference imaging device 17 is switchable between multiple imaging devices. Therefore, compared to a case where the reference imaging device 17 is not switchable, the design flexibility in deciding which imaging device to set as the reference imaging device 17 is increased.

[0139] The CPU 58 outputs a reference image 46D obtained by the soccer field 24 being imaged by the reference imaging device 17, and, provided that a virtual viewpoint image generation instruction 107 is given, uses the reference imaging device information 46E as a reference to generate a virtual viewpoint image 46 corresponding to the reference image 46D. C is generated. Therefore, a virtual viewpoint image 46C having the same field of view as the reference imaging device information 46E is generated. With this configuration, the difference in the field of view between the reference image 46D and the virtual viewpoint image 46C can be reduced compared to the case where a virtual viewpoint image 46C that does not correspond to the reference image 46D is generated.

[0140] In the above embodiment, an example of a configuration in which multiple imaging devices continuously transmit imaging device information is given, but the technology of this disclosure is not limited to this. For example, imaging device information may be continuously transmitted by a server (not shown). In this case, the server is an example of a "transmitting device" related to the technology of this disclosure. The server holds imaging device information for multiple imaging devices. The control unit 100 receives the imaging device information transmitted from the server and stores it in the memory 62.

[0141] Furthermore, in the above embodiment, multiple imaging devices continuously transmitted imaging device information, but multiple imaging devices may transmit imaging device information to the control unit 100 in response to a transmission request from the control unit 100. That is, the control unit 100 may request the transmission of imaging device information only from the reference imaging device 17, and in response to the transmission request, the reference imaging device 17 may transmit its own imaging device information as reference imaging device information 46E.

[0142] Furthermore, in the above embodiment, the imaging device information of multiple imaging devices was stored in the memory 62, but the control unit 100 may store only the imaging device information of the reference imaging device 17 as reference imaging device information 46E in the memory 62 or on the server. In this case, since the reference imaging device 17 changes according to the operation of the video producer, the control unit 100 updates the reference imaging device information 46E held in the memory 62 or on the server each time it acquires the reference imaging device identifier 35, that is, each time it acquires the reference imaging device information 46E. The control unit 100 acquires the reference imaging device information 46E held on the server or in the memory 62.

[0143] Furthermore, in the above embodiment, the control unit 100 acquired reference imaging device information 46E from imaging device information transmitted by multiple imaging devices, but the control unit 100 may also acquire reference imaging device information 46E based on reference video 46D. In this case, for example, the soccer field 24 is provided with markers indicating the distance from a reference position at predetermined distances. On the soccer field 24, for example, with the center of the soccer field 24 as the reference position (0,0), markers indicating the distance X in the longitudinal direction and the distance Y in the short direction of the soccer field 24 in the form (X,Y) are provided in a grid pattern at regular intervals. Also, on each of the four walls surrounding the soccer field 24, with the center of the longitudinal direction of the wall and the ground as the reference position (0,0), markers indicating the distance X in the horizontal direction and the distance Y in the vertical direction from the reference position in the form (X,Y) are provided in a grid pattern at regular intervals. The control unit 100 can identify the field of view of the reference imaging device 17 by detecting a marker from the reference image 46D, and can acquire reference imaging device information 46E geometrically based on the identified field of view.

[0144] In the above embodiment, an example of how the reference image 46D is generated by the information processing device 12 was described, but the technology of this disclosure is not limited thereto. The reference image 46D may be generated by a device other than the information processing device 12 (hereinafter referred to as the "image generation device").

[0145] While the display 78 of the smart device 14 is given as an example of a "display" related to the technology disclosed herein, various devices with displays, such as head-up displays, head-mounted displays, personal computers, and / or wearable terminals, can also be used as "displays" related to the technology disclosed herein, instead of the smart device 14.

[0146] Furthermore, although a soccer field 22 was used as an example in the above embodiment, this is merely one example, and any location where multiple imaging devices and multiple sound-collecting devices can be installed may be used, such as a baseball field, rugby field, curling rink, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater.

[0147] Furthermore, although the above embodiment illustrates a wireless communication method using a base station 20, this is merely one example, and the technology of this disclosure can also be applied to a wired communication method using cables.

[0148] Furthermore, although an unmanned aerial vehicle 27 was used as an example in the above embodiment, the technology of this disclosure is not limited thereto, and the imaging area may be imaged by an imaging device 18 suspended by a wire (for example, a self-propelled imaging device that can move along the wire).

[0149] Furthermore, although computers 50 and 70 have been used as examples in the above embodiments, the technology of this disclosure is not limited thereto. For example, devices including ASICs, FPGAs, and / or PLDs may be used instead of computers 50 and / or 70. Alternatively, a combination of hardware and software configurations may be used instead of computers 50 and / or 70.

[0150] Furthermore, although the information processing device program is stored in the storage 60 in the above embodiment, the technology of this disclosure is not limited thereto. For example, as shown in Figure 20, the information processing device program may be stored in any portable storage medium 200 such as an SSD or USB memory. In this case, the information processing device program stored in the storage medium 200 is installed in the computer 50, and the CPU 58 executes the information processing device processing according to the information processing device program.

[0151] Alternatively, the information processing device program may be stored in the memory of another computer or server device connected to the computer 50 via a communication network (not shown), and the information processing device program may be downloaded to the information processing device 12 in response to a request from the information processing device 12. In this case, the information processing device processing based on the downloaded information processing device program is executed by the CPU 58 of the computer 50.

[0152] Furthermore, although the above embodiment uses CPU 58 as an example, the technology of this disclosure is not limited to this, and a GPU may be used. Also, instead of CPU 58, multiple CPUs or a combination of CPU and GPU may be used. In other words, the processing on the information processing device side may be executed by a single processor or multiple processors that are physically separated. Also, instead of CPU 88, a GPU may be used, or multiple CPUs or a combination of CPU and GPU may be used, and various processes may be executed by a single processor or multiple processors that are physically separated.

[0153] The following types of processors can be used as hardware resources to execute processing on the information processing device side. For example, as mentioned above, a CPU is a general-purpose processor that functions as a hardware resource to execute processing on the information processing device side according to software, i.e., a program. Other processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to perform particular processing. All of these processors have built-in or connected memory, and all of them execute processing on the information processing device side by using this memory.

[0154] The hardware resources that execute the processing on the information processing device side are among these various processors. It may consist of one of these, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource that performs the processing on the information processing device side may be a single processor.

[0155] Examples of configurations using a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, as exemplified by client and server computers, and this processor functions as a hardware resource that executes information processing device-side processing. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that execute information processing device-side processing, on a single IC chip, as exemplified by SoCs. In this way, information processing device-side processing is realized using one or more of the above types of processors as hardware resources.

[0156] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices.

[0157] Furthermore, the processing on the information processing device 12 described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0158] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0159] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0160] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Equipped with a processor, The aforementioned processor, A reference imaging device identifier is obtained to identify one of the multiple original images obtained by imaging the imaging area using multiple imaging devices. A reference video is generated by superimposing information indicating the state of the object to be viewed onto an image obtained by capturing the imaging area using the reference imaging device identified by the reference imaging device identifier, and this video is then displayed on the terminal's display. While the aforementioned reference video is being displayed, an instruction is received to start generating a virtual viewpoint image. At the time the aforementioned instruction is received, information regarding the position, imaging direction, and field of view of the reference imaging device, identified by the reference imaging device identifier at the time the frame displayed on the display is obtained, is acquired. A virtual viewpoint image having the same viewpoint position, line of sight direction, and field of view as the reference imaging device is generated based on multiple images obtained by imaging the imaging area using multiple imaging devices. Information processing device.

2. The aforementioned processor, Acquire image quality information indicating the image quality of the aforementioned reference video, Subject to the above instructions, the image quality of the virtual viewpoint image is determined based on the image quality information. The information processing apparatus according to claim 1.

3. In the initial display frame after receiving the instruction, the display is made to display using the reference image included in the reference video and the virtual viewpoint image, and thereafter the virtual viewpoint image is displayed on the display. The information processing apparatus according to claim 1 or claim 2.

4. The processor gradually changes the ratio between the reference image and the virtual viewpoint image. The information processing apparatus according to claim 3.

5. Displaying the virtual viewpoint image is achieved by outputting the virtual viewpoint image to the display. The processor receives a change signal that continuously changes at least one of the viewpoint position, the line of sight direction, and the field of view in the virtual viewpoint image output to the display. An information processing apparatus according to any one of claims 1 to 4.

6. The reference imaging device is an imaging device capable of changing at least one of the position, imaging direction, and field of view. An information processing apparatus according to any one of claims 1 to 5.

7. The aforementioned reference imaging device is one of the plurality of imaging devices. An information processing apparatus according to any one of claims 1 to 6.

8. The reference imaging device is switchable among the multiple imaging devices. The information processing apparatus according to claim 7.

9. A method for operating an information processing device comprising a processor and a memory connected to or built into the processor, To obtain a reference imaging device identifier that identifies one of the multiple original images obtained by imaging a region using multiple imaging devices, A reference video is generated by superimposing information indicating the state of the object to be viewed onto an image obtained by capturing the imaging area using the reference imaging device identified by the reference imaging device identifier, and then displaying this reference video on the terminal's display. The system receives an instruction to start generating a virtual viewpoint image while the aforementioned reference video is being displayed. At the time the aforementioned instruction is received, information regarding the position, imaging direction, and field of view of the reference imaging device identified by the reference imaging device identifier at the time the frame displayed on the display is obtained is acquired, and, This includes generating a virtual viewpoint image having the same viewpoint position, line of sight direction, and field of view as the reference imaging device, based on multiple images obtained by imaging the imaging area with multiple imaging devices. How to operate an information processing device.

10. A computer applied to an information processing device comprising a processor and memory connected to or built into the processor, To obtain a reference imaging device identifier that identifies one of the multiple original images obtained by imaging a region using multiple imaging devices, A reference video is generated by superimposing information indicating the state of the object to be viewed onto an image obtained by capturing the imaging area using the reference imaging device identified by the reference imaging device identifier, and then displaying this reference video on the terminal's display. The system receives an instruction to start generating a virtual viewpoint image while the aforementioned reference video is being displayed. At the time the aforementioned instruction is received, information regarding the position, imaging direction, and field of view of the reference imaging device identified by the reference imaging device identifier at the time the frame displayed on the display is obtained is acquired, and, A program for performing a process that includes generating a virtual viewpoint image having the same viewpoint position, line of sight direction, and field of view as the reference imaging device, based on a plurality of images obtained by imaging the imaging area with a plurality of imaging devices.

Images