Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method and a program which acquire a virtual view point image showing a subject in a case where the subject is observed with virtual view point and visual line.SOLUTION: An information processing system being an information processing apparatus includes: a smart phone 14 having a detection unit 88A that detects a three-dimensional position and a posture of an object in an instruction three-dimensional region having an enlarged or reduced relation with an observation three-dimensional region in which a virtual viewpoint and a virtual visual line are defined; and an image generation device 12 having a derivation unit 58B that derives the viewpoint and the visual line corresponding to detection results of the detection unit depending on positional relation information indicating a relative positional relation between the observation three-dimensional region and the instruction three-dimensional region, and an acquisition unit 58C that acquires a virtual viewpoint image showing a subject in a case in which the subject is observed with the viewpoint and the visual line derived by the derivation unit, the virtual viewpoint image being based on a plurality of images obtained by imaging an imaging region included in the observation three-dimensional region by a plurality of imaging devices 16.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Japanese Patent Publication No. 2015-225529 discloses an information processing device that has an identification means for identifying a position and orientation of a terminal, a determination means for determining a position and orientation of a virtual camera based on the position and orientation of the terminal, a generation means for generating a free viewpoint image to be sent to the terminal based on the position and orientation of the virtual camera, a transmission means for transmitting the free viewpoint image generated by the generation means to the terminal, and a receiving means for receiving an instruction to change the settings of the virtual camera, wherein the generation means generates a free viewpoint image related to the change instruction when the change instruction is received.

[0003] Patent No. 6427258 discloses a display control device having an acquisition means for acquiring a plurality of virtual viewpoint images corresponding to the positions and attitudes of a plurality of virtual viewpoints, and a display control means for displaying the plurality of virtual viewpoint images acquired by the acquisition means on an operation screen for operation related to the position and attitude of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control means is an image generated based on a plurality of captured images obtained by capturing an image of an imaging target area by a plurality of cameras, and a virtual viewpoint image corresponding to a virtual viewpoint selected from the plurality of virtual viewpoints as an object for operation of position and attitude is transmitted to another device.

[0004] Japanese Patent Application Publication No. 2015-076062 discloses an image display device that displays images, comprising: an acquisition means for acquiring multiple images of a subject taken by multiple cameras; a generation means for combining the multiple images to generate a composite image of the subject viewed from a virtual viewpoint; a display control means for displaying the composite image on a screen; and a detection means for detecting a user operation for changing the position of the virtual viewpoint of the composite image displayed on the screen, wherein the generation means changes the position of the virtual viewpoint of the composite image based on the user operation.

[0005] JP 2018-092580 A discloses an image generation device that generates a virtual viewpoint image, which acquires information related to the installation of a group of cameras that capture images used to generate the virtual viewpoint image and virtual viewpoint information related to the virtual viewpoint, determines an image generation method to be used to generate the virtual viewpoint image based on the information related to the installation of the group of cameras and the virtual viewpoint information, and generates a virtual viewpoint image corresponding to the virtual viewpoint using the determined image generation method. Summary of the Invention

[0006] One embodiment of the technology disclosed herein provides an information processing device, an information processing method, and a program that can easily acquire a virtual viewpoint image showing a subject when observed from a position different from the actual observation position, compared to acquiring a virtual viewpoint image showing the subject when observed from the actual observation position. [Means for solving the problem]

[0007] A first aspect of the technology of the present disclosure is an information processing device that includes a detection unit that detects a three-dimensional position and orientation of an object within a three-dimensional instruction area that has an enlarged or reduced relationship with an observation three-dimensional area that defines a virtual viewpoint and line of sight, a derivation unit that derives a viewpoint and line of sight corresponding to a detection result of the detection unit in accordance with positional relationship information that indicates the relative positional relationship between the observation three-dimensional area and the instruction three-dimensional area, and an acquisition unit that acquires a virtual viewpoint image based on a plurality of images obtained by capturing images of imaging areas included in the observation three-dimensional area by a plurality of imaging devices, the virtual viewpoint image showing a subject when observed from the viewpoint and line of sight derived by the derivation unit.

[0008] A second aspect of the technique of the present disclosure is the information processing device according to the first aspect, in which the relative positional relationship is a relationship between a position of the observation three-dimensional area relative to a reference point and a position of the instruction three-dimensional area relative to the reference point.

[0009] A third aspect of the technology disclosed herein is an information processing device according to the second aspect, in which the positional relationship information is information in which coordinates indicating a position within a three-dimensional observation area with a reference point as the origin correspond to coordinates indicating a position within a three-dimensional instruction area with the reference point as the origin.

[0010] A fourth aspect of the technology disclosed herein is an information processing device according to the second aspect, in which the positional relationship information is information including the degree of difference between the distance between a position in the three-dimensional observation area and a reference point and the distance between a position in the three-dimensional instruction area and the reference point.

[0011] A fifth aspect of the technology disclosed herein is an information processing device relating to the second aspect, in which the reference points are classified into observation reference points applied to a three-dimensional area for observation and instruction reference points applied to a three-dimensional area for instruction, and the instruction reference points correspond to the observation reference points at positions different from the observation reference points.

[0012] A sixth aspect according to the technology of the present disclosure is the information processing device according to the fifth aspect, in which the indication reference point is determined by detecting a three-dimensional position by a detection unit.

[0013] A seventh aspect of the technology of the present disclosure is an information processing device relating to any one of the first to sixth aspects, wherein the detection unit detects the three-dimensional positions of at least three points, and further includes a generation unit that generates a three-dimensional instruction area based on an instruction reference surface identified by using the at least three three-dimensional positions detected by the detection unit.

[0014] An eighth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to sixth aspects, further including a generation unit that generates a three-dimensional area for instruction based on a reference surface for instruction identified by using an image obtained by imaging a reference subject.

[0015] A ninth aspect of the technology of the present disclosure is an information processing device according to any one of the first to sixth aspects, further including a generation unit that uses a display surface of a first three-dimensional area image for observation, obtained by imaging the three-dimensional area for observation by a first imaging device, or a first three-dimensional area image for observation obtained by imaging the three-dimensional area for observation by a first virtual imaging device, as a reference surface for instruction, and generates a three-dimensional area for instruction based on the reference surface.

[0016] A tenth aspect of the technique of the present disclosure is the information processing device according to the ninth aspect, in which the first observational three-dimensional area image is an image showing an observational three-dimensional area in a bird's-eye view.

[0017] An eleventh aspect of the technology of the present disclosure is an information processing device according to the ninth or tenth aspect, further including an image control unit that enlarges or reduces the first observation three-dimensional area image in accordance with given instructions.

[0018] A twelfth aspect of the technology of the present disclosure is an information processing device relating to any one of the seventh to eleventh aspects, in which the reference surface for instruction is a surface that defines one outer surface of the three-dimensional area for instruction, one outer surface of the three-dimensional area for instruction, or an internal surface of the three-dimensional area for instruction.

[0019] A thirteenth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to twelfth aspects, in which the detection unit detects the attitude by measuring the pitch angle, yaw angle, and roll angle of the object, and the acquisition unit acquires a virtual viewpoint image oriented according to the attitude detected by measuring the roll angle by the detection unit.

[0020] A 14th aspect of the technology of the present disclosure is an information processing device relating to any one of the first to thirteenth aspects, in which the observation three-dimensional area is a three-dimensional area defined based on a second observation three-dimensional area image obtained by imaging a specific reference surface by a second imaging device or obtained by imaging a specific reference surface by a second virtual imaging device, and has a similarity relationship with the instruction three-dimensional area, and includes a reception unit that receives a size change instruction to change the size of the specific reference surface, and a change unit that changes the size of the observation three-dimensional area by changing the size of the specific reference surface in accordance with the size change instruction received by the reception unit, and in which the similarity relationship is maintained regardless of the change in the size of the observation three-dimensional area.

[0021] A 15th aspect of the technology disclosed herein is an information processing device relating to any one of the first to thirteenth aspects, which includes a reception unit that receives an image size change instruction to enlarge or reduce a specific reference plane image while an image including a specific reference plane image indicating a specific reference plane is displayed by a second display device, and the positional relationship information is information including information indicating the relative positional relationship between a three-dimensional area in real space corresponding to the specific reference plane image enlarged or reduced in accordance with the image size change instruction received by the reception unit and the instruction three-dimensional area.

[0022] A sixteenth aspect of the technology of the present disclosure is an information processing device according to the fourteenth aspect, in which the change unit changes the size of the three-dimensional area for observation by enlarging or reducing the second three-dimensional area image for observation in accordance with a size change instruction.

[0023] A 17th aspect of the technology of the present disclosure is an information processing device relating to any one of the 14th to 16th aspects, in which the reception unit further receives an imaging direction change instruction to change the imaging direction of the second imaging device or the second virtual imaging device, and the change unit changes the imaging direction in accordance with the imaging direction change instruction received by the reception unit.

[0024] An 18th aspect of the technology of the present disclosure is an information processing device according to any one of the 1st to 17th aspects, further including an output unit that outputs the virtual viewpoint image acquired by the acquisition unit.

[0025] A 19th aspect of the technology of the present disclosure is an information processing method that includes detecting a three-dimensional position and orientation of an object within a three-dimensional instruction area that has an enlarged or reduced relationship to an observation three-dimensional area that defines a virtual viewpoint and line of sight, deriving a viewpoint and line of sight corresponding to the detection result in accordance with positional relationship information that indicates the relative positional relationship between the observation three-dimensional area and the instruction three-dimensional area, and obtaining a virtual viewpoint image based on multiple images obtained by capturing images of imaging areas included in the observation three-dimensional area by multiple imaging devices, the virtual viewpoint image showing the subject when observed from the derived viewpoint and line of sight.

[0026] A twentieth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including: detecting a three-dimensional position and orientation of an object within a three-dimensional instruction area that has an enlarged or reduced relationship with an observation three-dimensional area that defines a virtual viewpoint and line of sight; deriving a viewpoint and line of sight corresponding to the detection result in accordance with positional relationship information that indicates the relative positional relationship between the observation three-dimensional area and the instruction three-dimensional area; and acquiring a virtual viewpoint image based on multiple images obtained by capturing images of imaging areas included in the observation three-dimensional area by multiple imaging devices, the virtual viewpoint image showing the subject when observed from the derived viewpoint and line of sight. [Brief description of the drawings]

[0027] [Figure 1] 1 is a schematic perspective view showing an example of an external configuration of an information processing system according to an embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of a relationship between a three-dimensional observation space and a three-dimensional instruction space generated by an information processing system according to an embodiment. FIG. [Diagram 3]2 is a block diagram showing an example of a hardware configuration of an electrical system of an image generating apparatus according to an embodiment, and an example of a relationship between the image generating apparatus and peripheral devices. FIG. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of an electrical system of a smartphone and a tablet terminal according to an embodiment. [Diagram 5] 1 is a conceptual diagram showing an example of a manner in which the smartphone according to the embodiment is rotated around a roll axis. FIG. [Figure 6] 1 is a conceptual diagram showing an example of a manner in which the smartphone according to the embodiment is rotated around a yaw axis. FIG. [Figure 7] 1 is a conceptual diagram showing an example of a manner in which the smartphone according to the embodiment is rotated around a pitch axis. FIG. [Figure 8] 2 is a block diagram showing an example of main functions of an image generating device and a smartphone according to an embodiment; FIG. [Figure 9] 1A to 1C are conceptual diagrams illustrating an example of a method for generating a three-dimensional area for observation by an image generating device according to an embodiment. [Figure 10] 1 is a conceptual diagram showing an example of a mode in which a recording medium P is imaged by a smartphone according to an embodiment. [Figure 11] 11 is a conceptual diagram illustrating an example of a method for generating a three-dimensional instruction area by a generation unit of the smartphone according to the embodiment. FIG. [Figure 12] 10A and 10B are conceptual diagrams illustrating an example of a method for generating positional relationship information by a region matching unit of the image generating device according to the embodiment. [Figure 13] 1 is a conceptual diagram showing an example of a mode in which a three-dimensional instruction area according to an embodiment is used. [Figure 14] 11A to 11C are conceptual diagrams for explaining an example of a method for detecting a smartphone position and a smartphone orientation within a three-dimensional instruction area according to an embodiment. [Figure 15] 10A to 10C are conceptual diagrams illustrating an example of a method for deriving an observation viewpoint and an observation line of sight by a derivation unit of an image generating device according to an embodiment. [Figure 16]1 is a conceptual diagram illustrating an example of a method for acquiring a virtual viewpoint image by an acquisition unit of an image generating device according to an embodiment. [Figure 17] 16 is a conceptual diagram showing an example of a mode of use of an instruction three-dimensional area when realizing an observation viewpoint and an observation line of sight different from those shown in FIG. 15. [Figure 18] 18 is a conceptual diagram showing an example of a manner in which a virtual viewpoint image showing a subject when a three-dimensional observation area is observed from the observation viewpoint and observation line of sight shown in FIG. 17 is generated. [Figure 19] 11 is a state transition diagram showing an example of transition of a display mode of a virtual viewpoint image when the observation viewpoint and the observation line of sight are changed. [Figure 20] 11 is a conceptual diagram showing an example of a manner in which an instruction to a touch panel is accepted when enlarging or reducing an overhead image displayed on a display of a smartphone according to an embodiment. [Figure 21] 11A and 11B are conceptual diagrams illustrating an example of a manner in which an overhead image displayed on a display of a smartphone according to an embodiment is enlarged and reduced. [Figure 22] 2 is a conceptual diagram showing an example in which the imaging range of the imaging device of the unmanned aerial vehicle is narrowed compared to the example shown in FIG. 1. [Figure 23] 13 is a conceptual diagram showing an example of processing contents when the size of the instruction three-dimensional region is changed in conjunction with a change in the size of the observation three-dimensional region by a change unit of the image generating device according to the embodiment. FIG. [Figure 24] 10 is a block diagram illustrating an example of a process performed by a change rate indicator of the image generating device according to the embodiment; FIG. [Diagram 25] 11 is a flowchart illustrating an example of the flow of a generation process executed by the smartphone according to the embodiment. [Figure 26] 11 is a flowchart illustrating an example of the flow of a detection process executed by the smartphone according to the embodiment. [Figure 27] 10 is a flowchart showing an example of the flow of a region matching process executed by the image generating device according to the embodiment. [Figure 28]10 is a flowchart showing an example of the flow of a virtual viewpoint image generation process executed by the image generation device according to the embodiment. [Figure 29] 10 is a flowchart showing an example of the flow of an image control process executed by the image generating device according to the embodiment. [Diagram 30] 10 is a flowchart showing an example of the flow of a change process executed by the image generating device according to the embodiment. [Diagram 31] 11 is a flowchart showing an example of the flow of a change rate instruction process executed by the image generating device according to the embodiment. [Diagram 32] 11 is a conceptual diagram showing an example of a case in which a three-dimensional instruction area is generated based on an overhead image displayed on a tablet terminal according to an embodiment. FIG. [Diagram 33] 13 is a conceptual diagram showing an example of the correspondence between the instruction three-dimensional area and a three-dimensional area within the observation three-dimensional area when a specific reference plane image displayed on a display is enlarged. FIG. [Diagram 34] 13 is a conceptual diagram showing an example of the correspondence between the instruction three-dimensional area and a three-dimensional area within the observation three-dimensional area when the specific reference plane image displayed on the display is reduced. FIG. [Diagram 35] 33 is a conceptual diagram showing an example of the positional relationship between the instruction three-dimensional area and the finger when an observation viewpoint and an observation line of sight different from the example shown in FIG. 32 are instructed. [Diagram 36] 13 is a conceptual diagram illustrating a first modified example of a method for generating a pointing reference surface. FIG. [Figure 37] FIG. 37 is a conceptual diagram showing an example of an aspect in which a three-dimensional instruction area generated based on the instruction reference surface shown in FIG. 36 is used. [Figure 38] 13 is a conceptual diagram illustrating a second modified example of a method for generating a pointing reference surface. FIG. [Figure 39] 1A to 1C are conceptual diagrams illustrating an example of virtual viewpoint images generated before and after the smartphone according to the embodiment is rotated around a roll axis. [Diagram 40] FIG. 11 is a block diagram showing a modified example of the configuration of the information processing system according to the embodiment. [Diagram 41] 11 is a block diagram showing an example of a manner in which an image generating apparatus-side program according to an embodiment is installed in a computer of the image generating apparatus from a storage medium on which the image generating apparatus-side program is stored. FIG. [Diagram 42] 11 is a block diagram showing an example of a manner in which a smartphone-side program according to an embodiment is installed in a computer of the smartphone from a storage medium in which the smartphone-side program is stored. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0029] First, the terms used in the following description will be explained.

[0030] CPU is an abbreviation for "Central Processing Unit". RAM is an abbreviation for "Random Access Memory". DRAM is an abbreviation for "Dynamic Random Access Memory". SRAM is an abbreviation for "Static Random Access Memory". ROM is an abbreviation for "Read Only Memory". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". GPU is an abbreviation for "Graphics Processing Unit". LAN is an abbreviation for "Local Area Network". 3D is an abbreviation for "3 Dimension". USB is an abbreviation for "Universal Serial Bus". In addition, in the explanation of this specification, the meaning of "flat" includes not only a completely flat surface, but also an approximately flat surface that includes allowable errors in design and manufacturing.

[0031] 1, as an example, an information processing system 10 includes an image generating device 12, a smartphone 14, a plurality of imaging devices 16, an imaging device 18, and a wireless communication base station (hereinafter simply referred to as a "base station") 20. Here, the information processing system 10 is an example of an "information processing device" according to the technology of the present disclosure, the smartphone 14 is an example of an "object" according to the technology of the present disclosure, and the imaging device 18 is an example of a "first imaging device" and a "second imaging device" according to the technology of the present disclosure.

[0032] The imaging devices 16 and 18 are imaging devices having CMOS image sensors and equipped with optical zoom and digital zoom functions. Note that other types of image sensors, such as CCD image sensors, may be used instead of the CMOS image sensors.

[0033] The multiple imaging devices 16 are installed in the soccer field 22. Each of the multiple imaging devices 16 is arranged to surround the soccer field 24, and captures an area including the soccer field 24 as an imaging area. Here, an example in which each of the multiple imaging devices 16 is arranged to surround the soccer field 24 is given, but the technology of the present disclosure is not limited to this, and the arrangement of the multiple imaging devices 16 is determined according to the virtual viewpoint image to be generated. The multiple imaging devices 16 may be arranged to surround the entire soccer field 24, or may be arranged to surround a specific part. The imaging device 18 is installed on an unmanned aerial vehicle (e.g., a drone), and captures an area including the soccer field 24 as an imaging area in a state where it is viewed from above. The imaging area in the state where the area including the soccer field 24 is viewed from above refers to the imaging surface of the imaging device 18 with respect to the soccer field 24. Here, the imaging surface of the imaging device 18 with respect to the soccer field 24 is an example of a "specific reference surface" according to the technology of the present disclosure.

[0034] The image generating device 12 is installed in a server room 32. The multiple imaging devices 16 and the image generating device 12 are connected via a LAN cable 30, and the image generating device 12 controls the multiple imaging devices 16 and acquires images captured by each of the multiple imaging devices 16. Note that, although a connection using a wired communication method via the LAN cable 30 is illustrated here as an example, the present invention is not limited to this, and a connection using a wireless communication method may also be used.

[0035] In the soccer stadium 22, spectator seats 26 are provided so as to surround a soccer field 24, and spectators 28 are seated in the spectator seats 25. Each spectator 28 carries a smartphone 14.

[0036] The base station 20 transmits and receives various information to and from the image generation device 12, the smartphone 14, and the unmanned aerial vehicle 27 via radio waves. That is, the image generation device 12 is connected to the smartphone 14 and the unmanned aerial vehicle 27 via the base station 20 so as to be able to wirelessly communicate with them. The image generation device 12 wirelessly communicates with the unmanned aerial vehicle 27 via the base station 20 to control the unmanned aerial vehicle 27 and to obtain images obtained by being captured by the imaging device 18 from the unmanned aerial vehicle 27.

[0037] The image generation device 12 is a device equivalent to a server, and the smartphone 14 is a device equivalent to a client terminal for the image generation device 12. The image generation device 12 and the smartphone 14 wirelessly communicate with each other via the base station 20, whereby the smartphone 14 requests the image generation device 12 to provide various services, and the image generation device 12 provides the smartphone 14 with the services in response to the requests from the smartphone 14.

[0038] As an example, as shown in FIG. 2, the information processing system 10 includes a tablet terminal 34. The tablet terminal 34 is used by a spectator 28. Like the smartphone 14, the tablet terminal 34 is a device equivalent to a client terminal for the image generation device 12. The tablet terminal 34 is connected to the image generation device 12 via the base station 20 so as to be able to wirelessly communicate with the image generation device 12. The image generation device 12 and the tablet terminal 34 wirelessly communicate with each other via the base station 20, whereby the tablet terminal 34 requests the image generation device 12 to provide various services, and the image generation device 12 provides the tablet terminal 34 with the services requested by the tablet terminal 34.

[0039] The information processing system 10 generates a three-dimensional observation area 36 and a three-dimensional instruction area 38. Both the three-dimensional observation area 36 and the three-dimensional instruction area 38 are invisible three-dimensional areas. Therefore, the three-dimensional observation area 36 and the three-dimensional instruction area 38 are not visually perceived in real space. In the example shown in Fig. 2, both the three-dimensional observation area 36 and the three-dimensional instruction area 38 are formed in a rectangular parallelepiped shape.

[0040] The three-dimensional observation area 36 is a three-dimensional area that defines a virtual viewpoint 42 and line of sight 44. The viewpoint 42 and line of sight 44 are the viewpoint and line of sight of the spectator 28 when the spectator 28 hypothetically observes the soccer field 24 within the three-dimensional observation area 36.

[0041] In the example shown in FIG. 2, the three-dimensional observation area 36 has a surface corresponding to the soccer field 24 as one outer surface 24A (hereinafter referred to as the "soccer field equivalent surface 24A"), and is generated based on the soccer field equivalent surface 24A. The soccer field equivalent surface 24A is a surface corresponding to the imaging surface of the soccer field 24 by the imaging device 18, and is defined as the bottom surface of the three-dimensional observation area 36. The height of the three-dimensional observation area 36 is defined within a predetermined range (several tens of meters in the example shown in FIG. 2). The height of the three-dimensional observation area 36 is defined according to, for example, the area of ​​the bottom surface of the three-dimensional observation area 36 (the area of ​​the soccer field equivalent surface 24A in the example shown in FIG. 2). The height of the three-dimensional observation area 36 may be a fixed value, or may be a variable value that is changed within the above-mentioned predetermined range according to instructions given by the spectators 28, etc. Here, the “predefined range” is a range within which the viewpoint 42 and line of sight 44 can be set at a settable height, and is uniquely determined according to, for example, the installation positions, imaging ranges, and imaging directions of the imaging device 18 and each of the multiple imaging devices 16.

[0042] The instruction three-dimensional area 38 is a three-dimensional area in the shape of a rectangular parallelepiped, and is generated based on the instruction reference surface 40A. The instruction three-dimensional area 38 has a reduced relationship with the observation three-dimensional area 36. That is, the instruction three-dimensional area 38 is a three-dimensional area in which the observation three-dimensional area 36 is reduced at a preset magnification. Here, the "preset magnification" refers to, for example, "1 / 300". Here, "1 / 300" is exemplified as the preset magnification, but the technology of the present disclosure is not limited to this, and other reduced scales may be used.

[0043] The instruction reference surface 40A is one of the outer surfaces of the instruction three-dimensional area 38. In the example shown in FIG. 2, the instruction reference surface 40A forms the bottom surface of the instruction three-dimensional area 38. The instruction reference surface 40A is a plane specified by using a reference object image 41 (see FIG. 11), which is an image obtained by capturing an image of the reference object 40. In the example shown in FIG. 2, the reference object 40 is the recording medium P itself, and the recording medium P is a sheet of paper on whose surface an image showing a soccer field 24 (hereinafter also simply referred to as a "soccer field image") is formed without margins. The area of ​​the soccer field image is the area of ​​the soccer field equivalent surface 24A reduced at a preset magnification, and the soccer field image is an image that is in a similar relationship to the soccer field equivalent surface 24A.

[0044] Here, a sheet of paper with a soccer field image formed on its surface with no margins is exemplified as the recording medium P, but the technology of the present disclosure is not limited to this, and the recording medium P may be a sheet of paper with a soccer field image formed on its surface with margins. Furthermore, the recording medium P is not limited to paper, and may be any recording medium on which an image can be formed, such as a transparent or semi-transparent sheet.

[0045] Spectators 28 determine the position of viewpoint 42 and the direction of line of sight 44 by positioning a specific part of smartphone 14 (in the example shown in FIG. 2, subject-side lens 14A of imaging device 84 (see FIG. 4)) within three-dimensional instruction area 38. In the example shown in FIG. 2, the position of subject-side lens 14A of imaging device 84 (see FIG. 4) mounted on smartphone 14 corresponds to the position of viewpoint 42, and the optical axis direction of subject-side lens 14A, in other words, the imaging direction of imaging device 84, corresponds to the direction of line of sight 44. The imaging direction of imaging device 84 is identified from the attitude of smartphone 14.

[0046] The image generating device 12 generates a virtual viewpoint image 46. The virtual viewpoint image 46 is a virtual viewpoint image based on a plurality of images obtained by capturing an image of an imaging area included in the three-dimensional observation area 36 using a plurality of imaging devices 16 (see FIG. 1). Capturing an imaging area refers to, for example, capturing an image at an angle of view including the imaging area. An example of the virtual viewpoint image 46 is a moving image using a 3D polygon. The image generating device 12 generates a moving image using a 3D polygon by synthesizing a plurality of images obtained by capturing an image of an imaging area included in the three-dimensional observation area 36 using a plurality of imaging devices 16. The image generating device 12 generates a virtual viewpoint image corresponding to a case where the imaging area is observed from an arbitrary position and an arbitrary direction based on the moving image using the generated 3D polygon. In one embodiment according to the technology of the present disclosure, the image generating device 12 generates a virtual viewpoint image 46 showing a subject when the subject is observed from a viewpoint 42 and a line of sight 44. In other words, the virtual viewpoint image 46 refers to an image equivalent to an image obtained by a virtual imaging device (hereinafter also referred to as a "virtual imaging device") installed at the position of the viewpoint 42 capturing images in the direction of the line of sight 44.

[0047] The image generating device 12 transmits a virtual viewpoint image 46 to the smartphone 14 and the tablet terminal 34 via the base station 20. The smartphone 14 and the tablet terminal 34 receive the virtual viewpoint image 46 transmitted from the image generating device 12. As an example, as shown in FIG. 2, the tablet terminal 34 includes a display 34A. An example of the display 34A is a liquid crystal display. Note that the display 34A is not limited to a liquid crystal display, and other types of displays such as an organic EL display may be adopted. In the example shown in FIG. 2, the virtual viewpoint image 46 is displayed by the display 34A.

[0048] As an example, as shown in Fig. 3, the image generating device 12 includes a computer 50, a receiving device 52, a display 53, a first communication I / F 54, and a second communication I / F 56. The computer 50 includes a CPU 58, a storage 60, and a memory 62, and the CPU 58, the storage 60, and the memory 62 are connected via a bus line 64. In the example shown in Fig. 3, for convenience of illustration, one bus line is shown as the bus line 64, but the bus line 64 includes a data bus, an address bus, a control bus, and the like.

[0049] The CPU 58 controls the entire image generating device 12. The storage 60 stores various parameters and various programs. The storage 60 is a non-volatile storage device. Here, an EEPROM is used as an example of the storage 60, but the storage 60 is not limited to this and may be a mask ROM, HDD, SSD, or the like. The memory 62 is a volatile storage device. Various information is temporarily stored in the memory 62. The memory 62 is used as a work memory by the CPU 58. Here, a DRAM is used as an example of the memory 62, but the storage 60 is not limited to this and may be another type of volatile storage device such as an SRAM.

[0050] The receiving device 52 receives instructions from a user of the image generating apparatus 12. Examples of the receiving device 52 include a touch panel, a hard key, and a mouse. The receiving device 52 is connected to the bus line 64, and the instructions received by the receiving device 52 are acquired by the CPU 58.

[0051] The display 53 is connected to the bus line 64, and displays various information under the control of the CPU 58. An example of the display 53 is a liquid crystal display. Note that the display 53 is not limited to a liquid crystal display, and other types of displays such as an organic electroluminescence (EL) display may also be used.

[0052] The first communication I / F 54 is connected to the LAN cable 30. The first communication I / F 54 is realized by, for example, a device having an FPGA. The first communication I / F 54 is connected to the bus line 64, and controls the exchange of various information between the CPU 58 and the multiple imaging devices 16. For example, the first communication I / F 54 controls the multiple imaging devices 16 according to a request from the CPU 58. The first communication I / F 54 also acquires images obtained by imaging each of the multiple imaging devices 16, and outputs the acquired images to the CPU 58. In the example shown in FIG. 3, a moving image is shown as the image obtained by imaging the imaging device 16, and the first communication I / F 54 outputs the moving image acquired from the imaging device 16 to the CPU 58.

[0053] In the example shown in Figure 3, a moving image is shown as the image obtained by capturing an image using the imaging device 16, but this is not limited thereto, and the image obtained by capturing an image using the imaging device 16 may be a still image, as long as it is an image that can be used to generate a virtual viewpoint image 46 (see Figure 2).

[0054] The second communication I / F 56 is connected to the base station 20 so as to be able to communicate wirelessly. The second communication I / F 56 is realized, for example, by a device having an FPGA. The second communication I / F 56 is connected to a bus line 64. The second communication I / F 56 controls the exchange of various information between the CPU 58 and the unmanned aerial vehicle 27 via the base station 20 by wireless communication. In addition, the second communication I / F 56 controls the exchange of various information between the CPU 58 and the smartphone 14 by wireless communication via the base station 20. In addition, the second communication I / F 56 controls the exchange of various information between the CPU 58 and the tablet terminal 34 by wireless communication via the base station 20.

[0055] As an example, as shown in Fig. 4, the smartphone 14 includes a computer 70, an acceleration sensor 72, a gyro sensor 74, a receiving device 76, a display 78, a microphone 80, a speaker 82, an imaging device 84, and a communication I / F 86. The computer 70 includes a CPU 88, a storage 90, and a memory 92, and the CPU 88, the storage 90, and the memory 92 are connected via a bus line 94. In the example shown in Fig. 4, for convenience of illustration, one bus line is illustrated as the bus line 94, but the bus line 94 includes a data bus, an address bus, a control bus, and the like.

[0056] The CPU 88 controls the entire smartphone 14. The storage 90 stores various parameters and various programs. The storage 90 is a non-volatile storage device. Here, an EEPROM is used as an example of the storage 90, but the storage 90 is not limited to this and may be a mask ROM, HDD, SSD, or the like. The memory 92 is a volatile storage device. Various information is temporarily stored in the memory 92, and the memory 92 is used as a work memory by the CPU 88. Here, a DRAM is used as an example of the memory 92, but the storage 90 is not limited to this and may be another type of volatile storage device such as an SRAM.

[0057] The acceleration sensor 72 measures the acceleration (hereinafter also simply referred to as “acceleration”) of the smartphone 14. The acceleration sensor 72 is connected to a bus line 94, and acceleration information indicating the acceleration measured by the acceleration sensor 72 is acquired by the CPU 88 via the bus line 94.

[0058] The gyro sensor 74 measures the angle around the yaw axis of the smartphone 14 (hereinafter also referred to as the "yaw angle"), the angle around the roll axis of the smartphone 14 (hereinafter also referred to as the "roll angle"), and the angle around the pitch axis of the smartphone 14 (hereinafter also referred to as the "pitch angle"). The gyro sensor 74 is connected to a bus line 94, and angle information indicating the yaw angle, roll angle, and pitch angle measured by the gyro sensor 74 is acquired by the CPU 88 via the bus line 94. Note that the acceleration sensor 72 and the gyro sensor 74 may be implemented as an integrated multi-axis (e.g., 6-axis) sensor.

[0059] The reception device 76 is an example of a "reception section (reception device)" according to the technology of the present disclosure, and receives instructions from a user of the smartphone 14 (here, as an example, the spectator 28). Examples of the reception device 76 include a touch panel 76A and a hard key. The reception device 76 is connected to the bus line 94, and the instructions received by the reception device 76 are acquired by the CPU 88.

[0060] The display 78 is connected to the bus line 94, and displays various information under the control of the CPU 88. An example of the display 78 is a liquid crystal display. Note that the display 78 is not limited to a liquid crystal display, and other types of displays such as an organic electroluminescence (EL) display may also be used.

[0061] The smartphone 14 includes a touch panel display, which is realized by the touch panel 76A and the display 78. That is, the touch panel display is formed by overlapping the touch panel 76A with the display area of ​​the display 78.

[0062] The microphone 80 converts the collected sound into an electrical signal. The microphone 80 is connected to a bus line 94. The electrical signal obtained by converting the sound collected by the microphone 80 is acquired by the CPU 88 via the bus line 94.

[0063] The speaker 82 converts an electrical signal into sound. The speaker 82 is connected to a bus line 94. The speaker 82 receives an electrical signal output from the CPU 88 via the bus line 94, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the smartphone 14.

[0064] The imaging device 84 captures an image of a subject, thereby obtaining an image showing the subject. The imaging device 84 is connected to a bus line 94. The image obtained by capturing an image of the subject by the imaging device 84 is obtained by the CPU 88 via the bus line 94.

[0065] The communication I / F 86 is connected to the base station 20 so as to be able to communicate wirelessly. The communication I / F 86 is realized, for example, by a device having an FPGA. The communication I / F 86 is connected to a bus line 94. The communication I / F 86 is responsible for transmitting and receiving various types of information between the CPU 88 and an external device via the base station 20 by wireless communication. Here, examples of the "external device" include the image generating device 12, the unmanned aerial vehicle 27, and the tablet terminal 34.

[0066] Furthermore, the tablet terminal 34 has basically the same configuration as the smartphone 14. That is, the tablet terminal 34 includes a computer 100, an acceleration sensor 102, a gyro sensor 104, a receiving device 106, a touch panel 106A, a display 34A, a microphone 110, a speaker 112, an imaging device 114, a communication I / F 116, a CPU 118, a storage 120, a memory 122, and a bus line 124.

[0067] The computer 100 corresponds to the computer 70. The acceleration sensor 102 corresponds to the acceleration sensor 72. The gyro sensor 104 corresponds to the gyro sensor 74. The receiving device 106 corresponds to the receiving device 76. The touch panel 106A corresponds to the touch panel 76A. The display 34A corresponds to the display 78. The microphone 110 corresponds to the microphone 80. The speaker 112 corresponds to the speaker 82. The imaging device 114 corresponds to the imaging device 84. The communication I / F 116 corresponds to the communication I / F 86. The CPU 118 corresponds to the CPU 88. The storage 120 corresponds to the storage 90. The memory 122 corresponds to the memory 92. The bus line 124 corresponds to the bus line 94. The bus line 124 also includes a data bus, an address bus, a control bus, and the like, like the bus lines 64 and 94. The tablet terminal 34 differs from the smartphone 14 in that the size of the touch panel 106A is larger than the size of the touch panel 76A, and in that the size of the display 34A is larger than the size of the display 78.

[0068] As shown in FIG. 5 as an example, the roll axis is an axis passing through the center of the display 78 of the smartphone 14. The rotation angle of the smartphone 14 around the roll axis is measured by the gyro sensor 74 as the roll angle. As shown in FIG. 6 as an example, the yaw axis is an axis passing through the center of the side surface in the longitudinal direction of the side surface of the smartphone 14. The rotation angle of the smartphone 14 around the yaw axis is measured by the gyro sensor 74 as the yaw angle. As shown in FIG. 7 as an example, the pitch axis is an axis passing through the center of the side surface in the shorter direction of the side surface of the smartphone 14. The rotation angle of the smartphone 14 around the pitch axis is measured by the gyro sensor 74 as the pitch angle.

[0069] 8, in the smartphone 14, a generating program 90A and a detecting program 90B are stored in the storage 90. In the following, when it is not necessary to distinguish between the generating program 90A and the detecting program 90B, they will be referred to as "smartphone-side programs" without being assigned reference numerals.

[0070] The CPU 88 is an example of a "processor" according to the technology of the present disclosure, and the memory 92 is an example of a "memory" according to the technology of the present disclosure. The CPU 88 reads the smartphone-side program from the storage 90 and expands the read smartphone-side program in the memory 92. The CPU 88 transmits and receives various information to and from the image generating device 12 in accordance with the smartphone-side program expanded in the memory 92.

[0071] The CPU 88 reads out a generation program 90A from the storage 90, and loads the read generation program 90A in the memory 92. The CPU 88 operates as a generation unit 88B in accordance with the generation program 90A loaded in the memory 92. By operating as the generation unit 88B, the CPU 88 executes a generation process (see FIG. 25) described below. The CPU 88 also reads out a detection program 90B from the storage 90, and loads the read detection program 90B in the memory 92. The CPU 88 operates as a detection unit 88A in accordance with the detection program 90B loaded in the memory 92. By operating as the detection unit 88A, the CPU 88 executes a detection process (see FIG. 26) described below.

[0072] 8, in the image generating device 12, a storage 60 stores an area association program 60A, a virtual viewpoint image generating program 60B, an image control program 60C, a change program 60D, and a change speed instruction program 60E. In the following, when there is no need to distinguish between the area association program 60A, the virtual viewpoint image generating program 60B, the image control program 60C, the change program 60D, and the change speed instruction program 60E, they will be referred to as "image generating device side programs" without being assigned reference numerals.

[0073] The CPU 58 is an example of a "processor" according to the technology of the present disclosure, and the memory 62 is an example of a "memory" according to the technology of the present disclosure. The CPU 58 reads the image generating device-side program from the storage 60 and expands the read image generating device-side program in the memory 62. The CPU 58 transmits and receives various types of information to and from the smartphone 14, the imaging device 16, the unmanned aerial vehicle 27, and the tablet terminal 34 in accordance with the image generating device-side program expanded in the memory 62.

[0074] CPU 58 reads area matching program 60A from storage 60, and loads read area matching program 60A in memory 62. CPU 58 operates as area matching unit 58A in accordance with area matching program 60A loaded in memory 62. CPU 58 operates as area matching unit 58A to execute area matching processing (see FIG. 27), which will be described later.

[0075] The CPU 58 reads out the virtual visual point image generating program 60B from the storage 60, and loads the read out virtual visual point image generating program 60B in the memory 62. The CPU 58 operates as a derivation unit 58B, an acquisition unit 58C, and an output unit 58D in accordance with the virtual visual point image generating program 60B loaded in the memory 62. The CPU 58 operates as the derivation unit 58B, the acquisition unit 58C, and the output unit 58D, thereby executing a virtual visual point image generating process (see FIG. 28) described later.

[0076] CPU 58 reads image control program 60C from storage 60, and loads read image control program 60C in memory 62. CPU 58 operates as image control unit 58E in accordance with image control program 60C loaded in memory 62. CPU 58 operates as image control unit 58E to execute image control processing (see FIG. 29), which will be described later.

[0077] CPU 58 reads change program 60D from storage 60, and loads read change program 60D in memory 62. CPU 58 operates as change unit 58F in accordance with change program 60D loaded in memory 62. CPU 58 operates as change unit 58F to execute a change process (see FIG. 30) described below.

[0078] The CPU 58 reads out the change rate instruction program 60E from the storage 60, and loads the read change rate instruction program 60E in the memory 62. The CPU 58 operates as a change rate instruction unit 58G in accordance with the change rate instruction program 60E loaded in the memory 62. The CPU 58 operates as the change rate instruction unit 58G to execute a change rate instruction process (see FIG. 31) described below.

[0079] In the image generating device 12, the storage 60 stores three-dimensional observation area information 60F. The three-dimensional observation area information 60F is information indicating the three-dimensional observation area 36. In the example shown in FIG. 9, the soccer field equivalent surface 24A has an observation reference point 36A. The soccer field equivalent surface 24A is a flat surface formed in a rectangular shape. The observation reference point 36A is one of the four corners that define the soccer field equivalent surface 24A, and is a reference point applied to the three-dimensional observation area 36. The three-dimensional observation area information 60F refers to, for example, three-dimensional coordinates (hereinafter also referred to as "three-dimensional observation area coordinates") indicating a position within the three-dimensional observation area 36 with the observation reference point 36A as the origin.

[0080] The observation three-dimensional area information 60F is information defined based on an overhead image showing the soccer field 24 in a state of being viewed from above. Here, the "overhead image" is an example of the "first observation three-dimensional area image" and the "second observation three-dimensional area image" according to the technology of the present disclosure. The overhead image is obtained by imaging the soccer field 24 by the imaging device 18 in a state where the unmanned aerial vehicle 27 is viewing the soccer field 24 from above. The CPU 58 acquires the overhead image from the unmanned aerial vehicle 27, identifies the observation reference point 36A based on the acquired overhead image, and generates the soccer field equivalent surface 24A with the identified observation reference point 36A as one of the four corners. The CPU 58 generates the observation three-dimensional area information 60F based on the soccer field equivalent surface 24A. That is, the CPU 58 generates the observation three-dimensional area information 60F showing the observation three-dimensional area 36 with the soccer field equivalent surface 24A as the bottom surface. In this way, the three-dimensional observation area 36 is a three-dimensional area defined based on the overhead image. The CPU 58 stores the generated three-dimensional observation area information 60F in the storage 60.

[0081] 10 as an example, in order to define the instruction three-dimensional area 38 (see FIG. 2), the spectator 28 captures an image of the reference subject 40 by operating the smartphone 14. For example, when an instruction to start capturing images by the imaging device 16 (hereinafter also referred to as an "imaging start instruction") is received by the touch panel 76A with the subject-side lens 14A of the smartphone 14 facing the reference subject 40 from above the reference subject 40, the reference subject 40 is captured by the imaging device 16.

[0082] As an example, as shown in FIG. 11, in the smartphone 14, the detection unit 88A includes a four-point three-dimensional position detection unit 88A1. The detection unit 88A detects at least three three-dimensional positions in the instruction three-dimensional area 38. Here, the detection unit 88A detects four three-dimensional positions in the instruction three-dimensional area 38. Specifically, the detection unit 88A detects four three-dimensional positions in the instruction three-dimensional area 38 by using the four-point three-dimensional position detection unit 88A1. The four-point three-dimensional position detection unit 88A1 acquires a reference object image 41 obtained by capturing an image of the reference object 40 by the imaging device 84 from the imaging device 84. The four-point three-dimensional position detection unit 88A1 detects the positions of the four corners of the reference object 40 shown by the reference object image 41 acquired from the imaging device 84 as four three-dimensional positions, and generates four-point three-dimensional position information indicating a relative positional relationship between the positions of the detected four corners.

[0083] The generating unit 88B generates the three-dimensional instruction area 38 based on the instruction reference surface 40A identified by using the three-dimensional positions of at least three points detected by the detecting unit 88A. Here, the three-dimensional instruction area 38 is generated based on the instruction reference surface 40A identified by using the reference object image 41. The instruction reference surface 40A is identified by using the three-dimensional positions of four points detected from the reference object image 41.

[0084] In order to generate the instruction three-dimensional region 38, the generation unit 88B includes an observation three-dimensional region information acquisition unit 88B1, an instruction reference plane generation unit 88B2, and an instruction three-dimensional region generation unit 88B3.

[0085] The observation three-dimensional region information acquisition unit 88B1 requests the CPU 58 of the image generating device 12 to acquire the observation three-dimensional region information 60F from the storage 60 and transmit the observation three-dimensional region information 60F. The CPU 58 acquires the observation three-dimensional region information 60F from the storage 60 in response to the request from the observation three-dimensional region information acquisition unit 88B1, and transmits the acquired observation three-dimensional region information 60F to the observation three-dimensional region information acquisition unit 88B1. The observation three-dimensional region information acquisition unit 88B1 acquires the observation three-dimensional region information 60F by receiving it transmitted from the CPU 58.

[0086] The instruction reference plane generating unit 88B2 acquires four-point three-dimensional position information from the four-point three-dimensional position detecting unit 88A1, and generates the instruction reference plane 40A by using the acquired four-point three-dimensional position information. The instruction reference plane 40A is defined by the positions of the four corners of the reference subject 40, and is a plane equivalent to a plane obtained by reducing the soccer field equivalent plane 24A at a preset magnification.

[0087] The instruction three-dimensional area generating unit 88B3 acquires the observation three-dimensional area information 60F from the observation three-dimensional area information acquiring unit 88B1, and acquires the instruction reference plane 40A from the instruction reference plane generating unit 88B2. The instruction three-dimensional area generating unit 88B3 refers to the observation three-dimensional area information 60F and specifies one of the four corners of the instruction reference plane 40A that corresponds to the position of the observation reference point 36A as the instruction reference point 38A. The instruction reference point 38A is a reference point applied to the instruction three-dimensional area 38.

[0088] In the information processing system 10, the reference points for the observation three-dimensional area 36 and the instruction three-dimensional area 38 are classified into an observation reference point 36A and an instruction reference point 38A, and the instruction reference point 38A corresponds to the observation reference point 36A at a position different from the observation reference point 36A. In the following, when it is not necessary to distinguish between the observation reference point 36A and the instruction reference point 38A, they will be referred to simply as "reference points" without being given reference symbols.

[0089] The instruction three-dimensional area generating unit 88B3 references the observation three-dimensional area information 60F, the instruction reference surface 40A, and the positional relationship between the observation reference point 36A and the instruction reference point 38A, and generates a three-dimensional area having a reduced relationship with the observation three-dimensional area 36 that defines the virtual viewpoint and line of sight as the instruction three-dimensional area 38. Here, the "reduced relationship" refers to, for example, a relationship reduced by a preset magnification with respect to the observation three-dimensional area 36.

[0090] The instruction three-dimensional region 38 has the instruction reference point 38A as an origin corresponding to the observation reference point 36A, and is defined by the instruction three-dimensional region information 90C as a three-dimensional region having a similar relationship to the observation three-dimensional region 36. The instruction three-dimensional region information 90C is generated by the instruction three-dimensional region generating unit 88B3 based on the observation three-dimensional region information 60F, the instruction reference surface 40A, and the positional relationship between the observation reference point 36A and the instruction reference point 38A. The instruction three-dimensional region information 90C indicates, for example, a position in the instruction three-dimensional region 38 with the instruction reference point 38A as the origin, and indicates three-dimensional coordinates (hereinafter also referred to as "instruction three-dimensional region coordinates") that correspond to the observation three-dimensional region information 60F.

[0091] 12 as an example, the instruction three-dimensional region generating unit 88B3 stores the generated instruction three-dimensional region information 90C in the storage 90, and outputs the generated instruction three-dimensional region information 90C to the region associating unit 58A. The region associating unit 58A generates positional relationship information 60G indicating the relative positional relationship between the observation three-dimensional region 36 and the instruction three-dimensional region 38, and stores the generated positional relationship information 60G in the storage 60.

[0092] Here, the relative positional relationship refers to the relationship (relative relationship) between the position of the observation three-dimensional area 36 relative to the reference point and the position of the instruction three-dimensional area 38 relative to the reference point. The positional relationship information 60G is information in which the observation three-dimensional area information 60F and the instruction three-dimensional area information 90C are associated with each other. The information in which the observation three-dimensional area information 60F and the instruction three-dimensional area information 90C are associated with each other refers to information in which the observation three-dimensional area coordinates and the instruction three-dimensional area coordinates are associated one-to-one for positions corresponding to each other between the observation three-dimensional area 36 and the instruction three-dimensional area 38. The positional relationship information 60G may be such that the relative positional relationship between the observation three-dimensional area coordinates and the instruction three-dimensional area coordinates is associated by some means such as a table or an arithmetic expression.

[0093] By using the instruction three-dimensional area information 90C defined as described above, the smartphone 14 can detect the position and orientation of the smartphone 14 in the instruction three-dimensional area 38, as shown in FIG. 13 as an example. When the position and orientation of the smartphone 14 in the instruction three-dimensional area 38 is specified by the smartphone 14, the specified position and orientation of the smartphone 14 are notified to the image generating device 12 via the base station 20. Then, by using the positional relationship information 60G, the viewpoint 42 and line of sight 44 in the observation three-dimensional area 36 are specified by the image generating device 12 from the position and orientation of the smartphone 14 in the instruction three-dimensional area 38.

[0094] In order to enable the smartphone 14 to detect the position and attitude of the smartphone 14 in the three-dimensional area for instruction 38, as shown in FIG. 14 as an example, the audience 28 aligns the smartphone 14 with respect to the reference point for instruction 38A. In this case, the audience 28 operates the touch panel 76A with the subject-side lens 14A aligned with the reference point for instruction 38A, and the touch panel 76A receives alignment completion information indicating that alignment has been completed. As a result, the detection unit 88A detects that alignment of the smartphone 14 with respect to the reference point for instruction 38A has been completed. That is, the detection unit 88A detects the position of the smartphone 14 in the three-dimensional area for instruction 38 at the time when the alignment completion information is received by the touch panel 76A based on acceleration information from the acceleration sensor, and determines the detected position as the reference point for instruction 38A. The position of the smartphone 14 in the three-dimensional area for instruction 38 specifically refers to the three-dimensional position of the smartphone 14 in the three-dimensional area for instruction 38.

[0095] When the detection unit 88A detects that the alignment of the smartphone 14 with respect to the indication reference point 38A is completed, the detection unit 88A acquires angle information from the gyro sensor 74 and acceleration information from the acceleration sensor at a predetermined time interval (for example, at 0.05 second intervals). Note that the angle information and acceleration information acquired by the detection unit 88A are an example of the "detection result of the detection unit" according to the technology of the present disclosure.

[0096] The detection unit 88A acquires instruction three-dimensional area information 90C from the storage 90. Then, the detection unit 88A detects the three-dimensional position of the subject-side lens 14A in the instruction three-dimensional area 38 as the three-dimensional position of the smartphone 14 by using the acquired instruction three-dimensional area information 90C and the acceleration information acquired continuously from the time when the alignment is completed. In addition, the detection unit 88A detects the attitude of the smartphone 14 in the instruction three-dimensional area 38 by using the latest angle information acquired from the gyro sensor 74. Note that, hereinafter, the three-dimensional position of the subject-side lens 14A in the instruction three-dimensional area 38 is also referred to as the "smartphone position", and the attitude of the smartphone 14 in the instruction three-dimensional area 38 is also referred to as the "smartphone attitude".

[0097] As an example, as shown in FIG. 15, the detection unit 88A generates position and orientation information indicating the detected smartphone position and smartphone orientation, and outputs the generated position and orientation information to the derivation unit 58B. The derivation unit 58B derives the viewpoint 42 and line of sight 44 corresponding to the detection result of the detection unit 88A according to the positional relationship information 60G. In order to derive the viewpoint 42 and line of sight 44, the derivation unit 58B acquires the positional relationship information 60G from the storage 60 and acquires the position and orientation information from the detection unit 88A. Then, the derivation unit 58B derives the viewpoint 42 from the positional relationship information 60G and the position and orientation information. Specifically, the derivation unit 58B derives the viewpoint 42 in the observational three-dimensional area 36 by deriving the observational three-dimensional area information 60F corresponding to the smartphone position indicated by the position and orientation information from the positional relationship information 60G. Here, the derivation of the viewpoint 42 within the three-dimensional area 36 for observation specifically means the derivation of the position of the viewpoint 42 within the three-dimensional area 36 for observation.

[0098] Furthermore, the derivation unit 58B derives the line of sight 44 from the positional relationship information 60G and the position and orientation information. Specifically, the derivation unit 58B derives the line of sight 44 within the three-dimensional observation area 36 by deriving, from the positional relationship information 60G, three-dimensional observation area information 60F corresponding to the smartphone orientation indicated by the position and orientation information. Here, the derivation of the line of sight 44 within the three-dimensional observation area 36 specifically means the derivation of the direction of the line of sight 44 within the three-dimensional observation area 36.

[0099] In the following, the viewpoint 42 derived by the derivation unit 58B will also be referred to simply as the "observation viewpoint" without being given a reference number, and the line of sight 44 derived by the derivation unit 58B will also be referred to simply as the "observation line of sight" without being given a reference number.

[0100] As an example, as shown in FIG. 16, derivation unit 58B generates viewpoint line of sight information indicating the derived observing viewpoint and the derived observing line of sight, and outputs the generated viewpoint line of sight information to acquisition unit 58C.

[0101] The acquisition unit 58C acquires a virtual viewpoint image 46 when the subject is observed from the observation viewpoint and observation line of sight derived by the derivation unit 58B. The acquisition unit 58C includes a viewpoint line of sight information acquisition unit 58C1 and a virtual viewpoint image generation unit 58C2. The viewpoint line of sight information acquisition unit 58C1 acquires viewpoint line of sight information from the derivation unit 58B.

[0102] The virtual viewpoint image generating unit 58C2 acquires viewpoint line-of-sight information from the viewpoint line-of-sight information acquiring unit 58C1. The virtual viewpoint image generating unit 58C2 also acquires moving images from the multiple imaging devices 16, and acquires, as moving images, overhead images obtained by imaging the imaging device 18 from the unmanned aerial vehicle 27. Here, the moving images acquired from the multiple imaging devices 16 and the moving images acquired from the imaging device 18 are examples of "multiple images" according to the technology of the present disclosure.

[0103] Here, a moving image is exemplified, but the technology of the present disclosure is not limited to this, and may be a still image. In addition, here, an example is shown in which an overhead image obtained by imaging by the imaging device 18 is also acquired by the virtual viewpoint image generating unit 58C2 and used to generate the virtual viewpoint image 46, but the technology of the present disclosure is not limited to this. For example, the overhead image may not be used to generate the virtual viewpoint image 46, and only a plurality of images obtained by imaging by each of the plurality of imaging devices 16 may be acquired by the virtual viewpoint image generating unit 58C2 and used to generate the virtual viewpoint image 46 by the virtual viewpoint image generating unit 58C2. That is, the virtual viewpoint image may be generated only from images obtained by imaging by the plurality of imaging devices 16 without using images obtained from the imaging device 18 (for example, a drone). In addition, if an image obtained from the imaging device 18 (for example, a drone) is used, a more accurate virtual viewpoint image can be generated.

[0104] The virtual viewpoint image generating unit 58C2 generates a virtual viewpoint image 46 in a case where a subject in an observation three-dimensional area is observed from the observation viewpoint and observation line of sight indicated by the viewpoint line of sight information, based on the moving images acquired from the multiple imaging devices 16 and the moving images acquired from the imaging device 18. In this manner, the acquiring unit 58C acquires the virtual viewpoint image 46 by the virtual viewpoint image generating unit 58C2 generating the virtual viewpoint image 46.

[0105] The output unit 58D outputs the virtual viewpoint image 46 acquired by the acquisition unit 58C to the tablet terminal 34. Specifically, the virtual viewpoint image 46 acquired by the acquisition unit 58C refers to the virtual viewpoint image 46 generated by the virtual viewpoint image generation unit 58C2. The tablet terminal 34 receives the virtual viewpoint image 46 output by the output unit 58D. The virtual viewpoint image 46 received by the tablet terminal 34 is displayed on the display 34A.

[0106] Here, an example in which the virtual viewpoint image 46 is displayed on the display 34A is shown, but the technology of the present disclosure is not limited to this. For example, the output destination of the virtual viewpoint image 46 by the output unit 58D may be a device other than a tablet terminal. The device other than a tablet terminal may be any device that is communicably connected to the image generating device 12, and may be, for example, a home server or a personal computer installed in the home of the spectator 28, other than the smartphone 14, or a server or a personal computer installed in the residence of a person other than the spectator 28. The virtual viewpoint image 46 may be output to a host computer or the like that controls the image generating device 12. The virtual viewpoint image 46 may be output to a storage 60 of the image generating device 12, and the virtual viewpoint image 46 may be stored in the storage 60. The virtual viewpoint image 46 may be output to an external storage such as a USB memory or an external SSD.

[0107] As shown in FIG. 17 as an example, when the smartphone position is changed by the audience 28 in the instruction three-dimensional area 38, the observation viewpoint and the observation line of sight are also changed. In this case, as shown in FIG. 18 as an example, a virtual viewpoint image 46 showing a subject in a different direction from the virtual viewpoint image 46 shown in FIG. 16 is generated and displayed on the display 34A. In this way, the observation viewpoint and the observation line of sight are changed with the change in the smartphone position in the instruction three-dimensional area 38, so that the content of the generated virtual viewpoint image 46 also changes every time the observation viewpoint and the observation line of sight are changed. That is, with the change in the smartphone position in the instruction three-dimensional area 38, as shown in FIG. 19 as an example, the aspect of the virtual viewpoint image 46 displayed on the display 34A, that is, the size and orientation of the subject shown in the virtual viewpoint image 46, change.

[0108] In order to realize the enlarged or reduced display of the overhead image, as shown in FIG. 20 as an example, the touch panel 76A of the smartphone 14 is operated by the audience 28 or the like, and the zoom in / out start instruction information is received by the touch panel 76A. The zoom in / out start instruction information is information that instructs the start of zooming in or out of the overhead image. When the zoom in / out start instruction information is received by the touch panel 76A, the smartphone 14 requests the image generating device 12 to transmit the overhead image. In response to the request from the smartphone 14, the image generating device 12 obtains the latest overhead image from the imaging device 18 of the unmanned aerial vehicle 27, and transmits the obtained overhead image to the smartphone 14. The smartphone 14 receives the overhead image transmitted from the image generating device 12. The received overhead image is displayed on the display 78.

[0109] When an instruction to enlarge (hereinafter referred to as an "enlargement instruction") is received by touch panel 76A while an overhead image is displayed on display 78, the overhead image displayed on display 78 is enlarged. When an instruction to reduce (hereinafter referred to as a "reduction instruction") is received by touch panel 76A while an overhead image is displayed on display 78, the overhead image displayed on display 78 is reduced. Note that, hereinafter, when there is no need to distinguish between an enlargement instruction and a reduction instruction, they will be referred to as "enlargement / reduction instructions."

[0110] As an example, as shown in FIG. 20, an example of an enlargement instruction is a pinch out operation on touch panel 76A, and an example of a reduction instruction is a pinch in operation on touch panel 76A.

[0111] 21, an enlargement / reduction instruction received by touch panel 76A is transmitted to image control unit 58E of image generating device 12 by smartphone 14. Image control unit 58E enlarges or reduces the overhead image in accordance with the given instruction.

[0112] When image control unit 58E receives a zoom-in / zoom-out instruction transmitted from smartphone 14, it acquires the latest overhead image from unmanned aerial vehicle 27. When image control unit 58E receives a zoom-in instruction transmitted from smartphone 14, it enlarges the overhead image acquired from unmanned aerial vehicle 27, and transmits the enlarged overhead image obtained by zooming in on the overhead image to smartphone 14. When image control unit 58E receives a zoom-out instruction transmitted from smartphone 14, it reduces the overhead image acquired from unmanned aerial vehicle 27, and transmits the reduced overhead image obtained by zooming in on the overhead image to smartphone 14.

[0113] The smartphone 14 receives the enlarged overhead image transmitted from the image control unit 58E and displays the received enlarged overhead image on the display 78. The smartphone 14 also receives the reduced overhead image transmitted from the image control unit 58E and displays the received reduced overhead image on the display 78.

[0114] As an example, as shown in FIG. 22, when a size change instruction to change the size of the imaging surface of the imaging device 18 for the soccer field 24 is received by a spectator 28 or the like through the touch panel 76A of the smartphone 14, the size of the imaging surface of the imaging device 18 for the soccer field 24 is changed. As in the example shown in FIG. 20, size change instructions include a pinch out operation and a pinch in operation. A pinch out operation is an operation used when making the size of the imaging surface of the imaging device 18 for the soccer field 24 narrower than the current time, and a pinch in operation is an operation used when making the size of the imaging surface of the imaging device 18 for the soccer field 24 larger than the current time. In the example shown in FIG. 22, the altitude of the unmanned aerial vehicle 27 is lowered from the current time, so that the size of the imaging surface of the imaging device 18 for the soccer field 24 becomes smaller than the current time (the example shown in FIG. 1).

[0115] As an example, as shown in FIG. 23, a size change instruction received by the touch panel 76A of the smartphone 14 is transmitted to the change unit 58F by the smartphone 14. The change unit 58F receives the size change instruction from the smartphone 14. When the change unit 58F receives the size change instruction, the change unit 58F changes the size of the imaging surface of the imaging device 18 with respect to the soccer field 24 in accordance with the received size change instruction to change the size of the three-dimensional observation area 36. Specifically, when the change unit 58F receives the size change instruction from the smartphone 14, it derives imaging range information (for example, the size (for example, area) of the imaging surface of the imaging device 18 with respect to the soccer field 24) indicating an imaging range that matches the size of the three-dimensional observation area 36 indicated by the received size change instruction, and transmits the derived imaging range information to the unmanned aerial vehicle 27. The derivation of the imaging range information is realized, for example, by using an imaging range derivation table in which the size of the three-dimensional observation area 36 indicated by the size change instruction is associated with the imaging range. Here, the size (e.g., area) of the imaging surface of the imaging device 18 relative to the soccer field 24 is exemplified as the imaging range information, but this is not limited to this. For example, if the imaging surface of the soccer field 24 is rectangular, the imaging range information may be coordinates that can identify the four corners of the imaging surface of the imaging device 18 relative to the soccer field 24, or any information that can identify the size of the imaging surface relative to the soccer field 24.

[0116] Note that instead of the imaging range derivation table, an imaging range derivation formula may be used. The imaging range derivation formula refers to a formula that has the size of the three-dimensional observation area 36 indicated by the size change instruction as an independent variable and has the imaging range as a dependent variable.

[0117] The unmanned aerial vehicle 27 receives imaging range information from the change unit 58F, moves to a position where imaging device 18 can capture images within the imaging range indicated by the received imaging range information, and obtains an overhead image by having imaging device 18 capture images.

[0118] The change unit 58F acquires an overhead image from the unmanned aerial vehicle 27. The change unit 58F changes the size of the observation three-dimensional region 36 by changing the observation three-dimensional region information 60F based on the overhead image acquired from the unmanned aerial vehicle 27. That is, the change unit 58F uses the overhead image acquired from the unmanned aerial vehicle 27 to change the size of the soccer field equivalent surface 24A that corresponds to the imaging surface of the soccer field 24 captured by the imaging device 18, and changes the observation three-dimensional region 36 to a three-dimensional region having the soccer field equivalent surface 24A with the changed size as the bottom surface.

[0119] The change unit 58F changes the size of the instruction three-dimensional region 38 while maintaining the similarity relationship with the observation three-dimensional region 36 in conjunction with the change in the size of the observation three-dimensional region 36. That is, the change unit 58F changes the size of the instruction three-dimensional region 38 by changing the instruction three-dimensional region information 90C so as to maintain the similarity relationship with the observation three-dimensional region 36 in conjunction with the change in the size of the observation three-dimensional region 36. The change unit 58F may change the size of the observation three-dimensional region 36 by enlarging or reducing the overhead image by the image control unit 58E (see FIGS. 20 and 21). In this case, the surface of the soccer field 24 indicated by the above-mentioned enlarged overhead image or reduced overhead image is an example of the "specific reference surface" according to the technology of the present disclosure. In this case, too, the modification unit 58F changes the size of the instruction three-dimensional region 38 by modifying the instruction three-dimensional region information 90C in conjunction with the change in the size of the observation three-dimensional region 36 so as to maintain a similarity relationship with the observation three-dimensional region 36.

[0120] In this manner, the positional relationship information 60G is updated by the change unit 58F changing the observation three-dimensional region information 60F and the instruction three-dimensional region information 90C while maintaining the similarity relationship. When the positional relationship information 60G is updated, the change unit 58F acquires the instruction three-dimensional region information 90C from the positional relationship information 60G and transmits the acquired instruction three-dimensional region information 90C to the instruction three-dimensional region generation unit 88B3 of the smartphone 14. The instruction three-dimensional region generation unit 88B3 receives the instruction three-dimensional region information 90C from the change unit 58F and overwrites and saves the received instruction three-dimensional region information 90C in the storage 90, thereby updating the instruction three-dimensional region information 90C in the storage 90.

[0121] As an example, as shown in FIG. 24, when the size of the three-dimensional region for observation 36 and the size of the three-dimensional region for instruction 38 are changed, the change unit 58F outputs change degree information 60H1 indicating the degree of change in the size of the three-dimensional region for observation 36 and the size of the three-dimensional region for instruction 38 to the change rate indication unit 58G.

[0122] The storage 60 stores a change speed derivation table 60H. In the change speed derivation table 60H, change degree information 60H1 is associated with change speed instruction information 60H2 that indicates the speed at which the size of the virtual viewpoint image 46 changes. The change degree information 60H1 and the change speed instruction information 60H2 are associated with each other in such a relationship that the larger the degree of change in the size of the observation three-dimensional area 36 and the size of the instruction three-dimensional area 38, the slower the size of the virtual viewpoint image 46 changes. In other words, the change degree information 60H1 and the change speed instruction information 60H2 are associated with each other in such a relationship that the smaller the degree of change in the size of the observation three-dimensional area 36 and the size of the instruction three-dimensional area 38, the faster the size of the virtual viewpoint image 46 changes.

[0123] The change speed instruction unit 58G derives the change speed instruction information 60H2 corresponding to the change degree information 60H1 input from the change unit 58F from the change speed derivation table 60H, and outputs the derived change speed instruction information 60H2 to the virtual viewpoint image generation unit 58C2. The virtual viewpoint image generation unit 58C2 changes the size of the virtual viewpoint image 46 at the speed instructed by the change speed instruction information 60H2 input from the change speed instruction unit 58G. The virtual viewpoint image generation unit 58C2 generates the virtual viewpoint image 46 whose size changes at the speed instructed by the change speed instruction information 60H2, and outputs the generated virtual viewpoint image 46 to the output unit 58D according to the speed instructed by the change speed instruction information 60H2. The output unit 58D outputs the virtual viewpoint image 46 to the tablet terminal 34 according to the speed instructed by the change speed instruction information 60H2.

[0124] Next, the operation of the information processing system 10 will be described.

[0125] First, an example of the flow of the generation process executed by the CPU 88 of the smartphone 14 in accordance with the generation program 90A will be described with reference to Fig. 25. Note that the generation process is executed, for example, when an instruction to start execution of the generation process is accepted by the acceptance device 76 (see Fig. 4) of the smartphone 14. Also, for convenience of explanation, the description will be given on the premise that the reference subject image 41 is acquired by the four-point three-dimensional position detection unit 88A1, and four-point three-dimensional position information has already been generated by the four-point three-dimensional position detection unit 88A1 based on the acquired reference subject image 41.

[0126] In the generation process shown in FIG. 25, first, in step ST10, the observation three-dimensional region information acquisition unit 88B1 acquires the observation three-dimensional region information 60F from the image generation device 12 (see FIG. 11), and then the generation process proceeds to step ST12.

[0127] In step ST12, the instruction reference plane generating unit 88B2 acquires four-point three-dimensional position information from the four-point three-dimensional position detecting unit 88A1 (see FIG. 11), and then the generation process proceeds to step ST14.

[0128] In step ST14, the instruction reference plane generating unit 88B2 generates the instruction reference plane 40A based on the four-point three-dimensional position information acquired in step ST12 (see FIG. 11), and then the generation process proceeds to step ST16.

[0129] In step ST16, the instruction three-dimensional area generation unit 88B3 generates the instruction three-dimensional area 38 by generating instruction three-dimensional area information 90C based on the observation three-dimensional area information 60F acquired in step ST10 and the instruction reference surface 40A generated in step ST14 (see Figure 11), and then the generation process proceeds to step ST18.

[0130] In step ST18, the instruction three-dimensional region generating unit 88B3 stores the instruction three-dimensional region information 90C generated in step ST16 in the storage 90 (see FIG. 12), and then the generation process proceeds to step ST20.

[0131] In step ST20, the instruction three-dimensional region generating unit 88B3 outputs the instruction three-dimensional region information 90C generated in step ST16 to the region associating unit 58A of the image generating device 12 (see FIG. 12), and then the generation process ends.

[0132] Next, an example of the flow of the detection process executed by the CPU 88 of the smartphone 14 in accordance with the detection program 90B will be described with reference to Fig. 26. Note that the detection process is executed when, for example, an instruction to start the execution of the detection process is accepted by the acceptance device 76 of the smartphone 14.

[0133] In the detection process shown in Fig. 26, first, in step T40, the detection unit 88A determines whether or not alignment completion information has been accepted by the accepting device 76 (see Fig. 14). In step ST40, if alignment completion information has been accepted by the accepting device 76 (see Fig. 4), the determination is affirmative, and the generation process proceeds to step ST42. In step ST40, if alignment completion information has not been accepted by the accepting device 76, the determination is negative, and the determination of step ST40 is performed again.

[0134] In step ST42, the detection unit 88A acquires angle information from the gyro sensor 74 (see FIG. 14), and then the detection process proceeds to step ST44.

[0135] In step ST44, the detection unit 88A acquires acceleration information from the acceleration sensor 72 (see FIG. 14), and then the detection process proceeds to step ST46.

[0136] In step ST46, the detection unit 88A detects the smartphone position based on the acceleration information acquired in step ST44, and detects the smartphone attitude based on the angle information acquired in step ST42, and then the detection process proceeds to step ST48.

[0137] In step ST48, the detection unit 88A generates position and orientation information indicating the smartphone position and smartphone orientation detected in step ST46, and outputs the generated position and orientation information to the derivation unit 58B of the image generation device 12 (see Figure 15), and then the detection processing proceeds to step ST50.

[0138] In step ST50, the detection unit 88A determines whether or not a condition for terminating the detection process (hereinafter referred to as a "detection process termination condition") has been satisfied. An example of the detection process termination condition is that an instruction to terminate the detection process has been accepted by the acceptance device 76. If the detection process termination condition has not been satisfied in step ST50, the determination is negative, and the detection process proceeds to step ST42. If the detection process termination condition has been satisfied in step ST50, the determination is positive, and the detection process ends.

[0139] Next, an example of the flow of the region matching process executed by the CPU 58 of the image generating device 12 in accordance with the region matching program 60A will be described with reference to Fig. 27. Note that the region matching process is executed on the condition that the execution of the generation process shown in Fig. 25 has ended, for example. In the following description, it is assumed that the three-dimensional region information for observation 60F has already been stored in the storage 60 of the image generating device 12.

[0140] In the region matching process shown in FIG. 27, first, in step ST60, the region matching section 58A acquires the three-dimensional region information for observation 60F from the storage 60, and then the region matching process proceeds to step ST62.

[0141] In step ST62, the region association section 58A acquires the instruction three-dimensional region information 90C from the instruction three-dimensional region generation section 88B3 (see FIG. 12), and then the detection process proceeds to step ST64.

[0142] In step ST64, the region association unit 58A generates positional relationship information 60G by associating the observation three-dimensional region information 60F acquired in step ST60 and the instruction three-dimensional region information 90C acquired in step ST62 between corresponding positions in the observation three-dimensional region 36 and the instruction three-dimensional region 38. Then, the region association unit 58A stores the generated positional relationship information 60G in the storage 60, and then the region association process ends.

[0143] Next, an example of the flow of the virtual viewpoint image generation process executed by the CPU 58 of the image generation device 12 in accordance with the virtual viewpoint image generation program 60B will be described with reference to Fig. 28. Note that the virtual viewpoint image generation process is executed when an instruction to start the execution of the virtual viewpoint image generation process is accepted by the acceptance device 52 of the image generation device 12 (see Fig. 3) or the acceptance device 76 of the smartphone 14 (see Fig. 4), for example.

[0144] In the virtual visual point image generating process shown in FIG. 28, first, in step ST80, the derivation unit 58B acquires the position and orientation information from the detection unit 88A (see FIG. 15), and then the virtual visual point image generating process proceeds to step ST82.

[0145] In step ST82, the derivation unit 58B acquires the positional relationship information 60G from the storage 60 (see FIG. 15), and thereafter, the virtual viewpoint image generating process proceeds to step ST84.

[0146] In step ST84, the derivation unit 58B derives the observation viewpoint and observation line of sight corresponding to the position and orientation information acquired in step ST80 in accordance with the positional relationship information 60G acquired in step ST82, and generates viewpoint line of sight information based on the derived observation viewpoint and observation line of sight, after which the virtual viewpoint image generation process proceeds to step ST86.

[0147] In step ST86, viewpoint line-of-sight information acquisition section 58C1 acquires viewpoint line-of-sight information from derivation section 58B (see FIG. 16), and thereafter, the virtual viewpoint image generation processing proceeds to step ST88.

[0148] In step ST88, the virtual viewpoint image generating section 58C2 acquires moving images from the imaging device 18 and each of the plurality of imaging devices 16, and then the virtual viewpoint image generating process proceeds to step ST90.

[0149] In step ST90, the virtual viewpoint image generation unit 58C2 generates a virtual viewpoint image 46 based on the observation viewpoint indicated by the viewpoint line of sight information acquired in step ST86, the observation line of sight indicated by the viewpoint line of sight information acquired in step ST86, and the moving image acquired in step ST88 (see Figure 16), and then the virtual viewpoint image generation process proceeds to step ST92.

[0150] In step ST92, the output unit 58D acquires the virtual viewpoint image 46 generated by the virtual viewpoint image generation unit 58C2 and outputs the acquired virtual viewpoint image 46 to the tablet terminal 34, and then the virtual viewpoint image generation process proceeds to step ST94.

[0151] In step ST94, the CPU 58 judges whether or not a condition for terminating the virtual viewpoint image generation process (hereinafter referred to as a "virtual viewpoint image generation process termination condition") is satisfied. An example of the virtual viewpoint image generation process termination condition is that an instruction to terminate the virtual viewpoint image generation process is accepted by the acceptance device 76 of the smartphone 14 or the acceptance device 52 of the image generation device 12. In step ST94, if the virtual viewpoint image generation process termination condition is not satisfied, the judgment is negative, and the virtual viewpoint image generation process proceeds to step ST80. In step ST94, if the virtual viewpoint image generation process termination condition is satisfied, the judgment is positive, and the virtual viewpoint image generation process ends.

[0152] Next, an example of the flow of the image control process executed by the CPU 58 of the image generating device 12 according to the image control program 60C will be described with reference to Fig. 29. Note that the image control process is executed when an instruction to start the execution of the image control process is accepted, for example, by the acceptance device 76 of the smartphone 14 or the acceptance device 52 of the image generating device 12. The following description is based on the assumption that the image control unit 58E has already acquired an overhead image from the unmanned aerial vehicle 27.

[0153] In the image control process shown in Fig. 29, first, in step ST100, the image control unit 58E determines whether or not a zoom-in / zoom-out instruction transmitted from the smartphone 14 has been received. In step ST100, if a zoom-in / zoom-out instruction transmitted from the smartphone 14 has not been received, the determination is negative, and the image control process proceeds to step ST104. In step ST100, if a zoom-in / zoom-out instruction transmitted from the smartphone 14 has been received, the determination is positive, and the image control process proceeds to step ST102.

[0154] In step ST102, the image control unit 58E enlarges or reduces the overhead image according to the enlargement / reduction instruction received in step ST100, and then the image control process proceeds to step ST104. That is, in step ST102, when the image control unit 58E receives an enlargement instruction, the overhead image is enlarged to generate an enlarged overhead image, and when the image control unit 58E receives a reduction instruction, the overhead image is reduced to generate a reduced overhead image. The enlarged overhead image or reduced overhead image generated in this manner is displayed on the display 78 of the smartphone 14 (see FIG. 21).

[0155] In step ST104, the CPU 58 determines whether or not a condition for terminating the image control processing (hereinafter referred to as an "image control processing termination condition") has been satisfied. An example of the image control processing termination condition is that an instruction to terminate the image control processing has been accepted by the acceptance device 76 of the smartphone 14 or the acceptance device 52 of the image generation device 12. In step ST104, if the image control processing termination condition is not satisfied, the determination is negative, and the image control processing proceeds to step ST100. In step ST104, if the image control processing termination condition is satisfied, the determination is positive, and the image control processing ends.

[0156] Next, an example of the flow of the change process executed by the CPU 58 of the image generating device 12 in accordance with the change program 60D will be described with reference to Fig. 30. Note that the change process is executed when an instruction to start execution of the change process is accepted by the acceptance device 76 of the smartphone 14 or the acceptance device 52 of the image generating device 12, for example.

[0157] In the change processing shown in FIG. 30, first, in step ST120, the change section 58F acquires an overhead image from the unmanned aerial vehicle 27, and then the change processing proceeds to step ST122.

[0158] In step ST122, the change unit 58F changes the size of the soccer field-equivalent surface 24A, which corresponds to the imaging surface of the soccer field 24 captured by the imaging device 18, based on the overhead image acquired in step ST120. Then, the change unit 58F changes the size of the three-dimensional observation area 36 by changing the observation three-dimensional area information 60F based on the soccer field-equivalent surface 24A whose size has been changed, and then the change process proceeds to step ST124.

[0159] In step ST124, the modification unit 58F modifies the instruction three-dimensional region information 90C based on the observation three-dimensional region information 60F modified in step ST122 so that the similarity relationship between the observation three-dimensional region 36 and the instruction three-dimensional region 38 is maintained, and then the modification process proceeds to step ST126.

[0160] In step ST126, the change section 58F outputs the instruction three-dimensional region information 90C obtained by the change in step ST124 to the instruction three-dimensional region generating section 88B3 of the smartphone 14 (see FIG. 23), and then the change process ends.

[0161] Next, an example of the flow of a change rate instruction process executed by the CPU 58 of the image generating device 12 in accordance with the change rate instruction program 60E will be described with reference to Fig. 31. Note that the change rate instruction process is executed, for example, when an instruction to start execution of the change rate instruction process is accepted by the acceptance device 76 of the smartphone 14 or the acceptance device 52 of the image generating device 12, and the size of the three-dimensional observation area 36 is changed by the change unit 58F.

[0162] In the change rate instruction process shown in FIG. 31, first, in step ST140, the change rate instruction section 58G acquires the change degree information 60H1 from the change section 58F, and then the change rate instruction process proceeds to step ST142.

[0163] In step ST142, the change rate instruction section 58G acquires the change rate derivation table 60H from the storage 60, and thereafter, the change rate instruction process proceeds to step ST144.

[0164] In step ST144, the change speed instruction section 58G derives change speed instruction information 60H2 corresponding to the change degree information 60H1 acquired in step ST140, from the change speed derivation table 60H acquired in step ST142. Then, the change speed instruction section 58G outputs the derived change speed instruction information 60H2 to the virtual viewpoint image generation section 58C2, and then the change speed instruction process proceeds to step ST146.

[0165] In step ST146, the virtual viewpoint image generating section 58C2 changes the size of the virtual viewpoint image 46 at the rate instructed by the change rate instruction information 60H2 input from the change rate instructing section 58G, and then the change rate instruction process ends.

[0166] As described above, in the information processing system 10, the detection unit 88A detects the smartphone position and smartphone orientation in the instruction three-dimensional area having a reduced relationship with the observation three-dimensional area 36. In addition, the derivation unit 58B derives the observation viewpoint and observation line of sight corresponding to the detection result of the detection unit 88A according to the positional relationship information 60G. Then, the acquisition unit 58C acquires the virtual viewpoint image 46 as an image showing the subject when the subject is observed from the observation viewpoint and observation line of sight derived by the derivation unit 58B. Therefore, compared to the case where a virtual viewpoint image showing the subject when the subject is observed from the actual observation position is acquired, it is possible to obtain a virtual viewpoint image (in the example shown in the above embodiment, the virtual viewpoint image 46) showing the subject when the subject is observed from a position different from the actual observation position.

[0167] Furthermore, in the information processing system 10, the relative positional relationship indicated by the positional relationship information 60G is the relationship between the position of the observation three-dimensional area 36 relative to the reference point and the position of the instruction three-dimensional area 38 relative to the reference point. Therefore, compared to the case where there is no reference point, the relationship between the position of the observation three-dimensional area 36 and the position of the instruction three-dimensional area 38 can be easily specified.

[0168] In the information processing system 10, the positional relationship information 60G is information in which coordinates indicating a position in the observation three-dimensional area 36 with the reference point as the origin correspond to coordinates indicating a position in the instruction three-dimensional area 38 with the reference point as the origin. Therefore, compared to a case in which coordinates indicating a position in the observation three-dimensional area 36 with the reference point as the origin are not associated with coordinates indicating a position in the instruction three-dimensional area 38 with the reference point as the origin, it is possible to easily identify the correspondence between the coordinates indicating a position in the observation three-dimensional area 36 and the coordinates indicating a position in the instruction three-dimensional area.

[0169] Furthermore, in the information processing system 10, the reference points are classified into an observation reference point 36A and an instruction reference point 38A, and the instruction reference point 38A corresponds to the observation reference point 36A at a position different from that of the observation reference point 36A. Therefore, compared to the case where a common reference point is applied to the observation three-dimensional area 36 and the instruction three-dimensional area 38, the relationship between the position in the observation three-dimensional area 36 and the instruction three-dimensional area 38 can be specified with high accuracy.

[0170] In the information processing system 10, the instruction reference point 38A is determined by detecting the smartphone position in the instruction three-dimensional area 38 by the detection unit 88A. Therefore, the instruction reference point 38A can be positioned at a position intended by a user such as a spectator 28 (hereinafter simply referred to as a "user").

[0171] Furthermore, in the information processing system 10, the four-point three-dimensional position detection unit 88A1 detects the three-dimensional positions of four points in the instruction three-dimensional area 38. Then, the generation unit 88B generates the instruction three-dimensional area 38 based on the instruction reference surface 40A identified by using the four three-dimensional positions detected by the four-point three-dimensional position detection unit 88A1. Therefore, it is possible to generate the instruction three-dimensional area 38 that reflects the position and size intended by the user.

[0172] Furthermore, in the information processing system 10, the generating unit 88B generates the instruction three-dimensional area 38 based on the instruction reference surface 40A specified by using the reference object image 41 obtained by capturing an image of the reference object 40. Therefore, it is possible to generate the instruction three-dimensional area 38 that allows the relationship with the reference object 40 to be understood.

[0173] Furthermore, in the information processing system 10, the overhead image is enlarged or reduced in size by the image control unit 58E in accordance with a zoom-in or zoom-out instruction. Therefore, the user can visually recognize the overhead image of the size intended by the user.

[0174] In the information processing system 10, the instruction reference surface 40A is one of the outer surfaces of the instruction three-dimensional area 38. Therefore, the instruction three-dimensional area 38 can be generated more quickly than when one surface selected by the user from a plurality of surfaces different from the instruction reference surface 40A is adopted as one of the outer surfaces of the instruction three-dimensional area 38.

[0175] In the information processing system 10, the observation three-dimensional area 36 is a three-dimensional area defined based on an overhead image obtained by capturing an image of an area including the soccer field 24 by the imaging device 18 of the unmanned aerial vehicle 27, and has a similarity relationship with the instruction three-dimensional area 38. Then, in accordance with a size change instruction received by the reception device 76 of the smartphone 14, the change unit 58F changes the size of the imaging surface of the imaging device 18 relative to the soccer field 24, thereby changing the size of the observation three-dimensional area 36. The change unit 58F changes the size of the instruction three-dimensional area 38 in a state in which the similarity relationship is indicated in conjunction with the change in the size of the observation three-dimensional area 36. Therefore, the change in the size of the observation three-dimensional area 36 can be reflected in the instruction three-dimensional area 38.

[0176] Furthermore, in the information processing system 10, the virtual viewpoint image 46 acquired by the acquisition unit 58C is output by the output unit 58D to the tablet terminal 34. Therefore, the virtual viewpoint image 46 acquired by the acquisition unit 58C can be provided to the user.

[0177] In the above embodiment, the width of the imaging range is changed by changing the altitude of the unmanned aerial vehicle 27, but the technology of the present disclosure is not limited to this. For example, the change unit 58F may change the width of the imaging range for the soccer field 24 (see Figures 2, 24, and 25, etc.) by enlarging or reducing the overhead image according to a size change instruction. This makes it possible to obtain a three-dimensional observation area 36 of a size corresponding to the imaging range intended by the user. The enlargement or reduction of the overhead image may be achieved by, for example, activating a digital zoom function or an optical zoom function of the imaging device 18 of the unmanned aerial vehicle 27.

[0178] In the above embodiment, the image obtained by capturing an image of the three-dimensional observation area 36 by the imaging device 18 of the unmanned aerial vehicle 27 is the bird's-eye view image, but the technology of the present disclosure is not limited to this. For example, a virtual viewpoint image obtained by capturing an image of the area including the soccer field 24, i.e., the three-dimensional observation area 36, ​​by a virtual camera may be the bird's-eye view image. Here, the "virtual viewpoint image captured by the virtual camera" refers to a virtual viewpoint image showing an area including the soccer field 24 when the soccer field 24 is observed from the sky. The virtual viewpoint image captured by the virtual camera is generated based on a plurality of images obtained by capturing an image of the area including the soccer field 24 by the imaging device 18 and a plurality of imaging devices 16 (see FIG. 1). Here, the "virtual camera" is an example of the "first virtual imaging device" and the "second virtual imaging device" according to the technology of the present disclosure.

[0179] In the above embodiment, the three-dimensional observation area 36 is captured by the imaging device 18 of the unmanned aerial vehicle 27 from above the soccer field 24, but the technology of the present disclosure is not limited to this. For example, the reception device 52 of the image generating device 12 or the reception device 76 of the smartphone 14 may receive an imaging direction change instruction to change the imaging direction of the imaging device 18, and the imaging direction of the imaging device 18 may be changed by the change unit 58F according to the received imaging direction change instruction. In addition, the imaging direction of the virtual camera may be changed by the change unit 58F according to the received imaging direction change instruction. In addition, when the reception device 52 of the image generating device 12 or the reception device 76 of the smartphone 14 receives an imaging direction change instruction, the imaging direction may be changed by using, instead of the imaging device 18, one of the multiple imaging devices 16 that corresponds to the imaging direction indicated by the imaging direction change instruction. By changing the imaging direction in this way, the three-dimensional observation area 36 when viewed from the direction intended by the user can be obtained.

[0180] In the above embodiment, the three-dimensional instruction area 38 is formed on the recording medium P having a soccer field image formed on its surface, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 32, the three-dimensional instruction area 38 may be formed on the display 34A of the tablet terminal 34 instead of the recording medium P.

[0181] In this case, the display surface of the display 34A in a state where an overhead image obtained by imaging the three-dimensional observation area 36 with the imaging device 18 is displayed on the display 34A is used as the instruction reference surface 40B. Then, the instruction three-dimensional area 38 is generated by the CPU 118 of the tablet terminal 34 based on the instruction reference surface 40B. The CPU 118 is an example of a "generator" according to the technology of the present disclosure. Here, an overhead image obtained by imaging the three-dimensional observation area 36 with the imaging device 18 is exemplified, but the present disclosure is not limited thereto, and a virtual viewpoint image obtained by imaging the three-dimensional observation area 36 with the virtual camera may be used as the overhead image.

[0182] 32, a finger cot 150 is attached to the finger of spectator 28, and the finger on which the finger cot 150 is attached (hereinafter simply referred to as the "finger") is imaged by imaging device 114 within three-dimensional instruction area 38. Then, an observation viewpoint and an observation line of sight are determined by CPU 118 based on an image obtained by imaging the finger by imaging device 114. In other words, the three-dimensional position of the finger within three-dimensional instruction area 38 is recognized by CPU 118 as a position corresponding to the position of the observation viewpoint, and the pointing direction of the finger within three-dimensional instruction area 38 is recognized by CPU 118 as a direction corresponding to the observation line of sight.

[0183] Here, a finger wearing a finger cot 150 is illustrated as an example, but the technology of the present disclosure is not limited to this, and a finger not wearing a finger cot 150 may be used. A finger is an example of an "object" according to the technology of the present disclosure. Here, the "object" can be rephrased as an "indicator." Here, a finger is illustrated as an example of an "object" according to the technology of the present disclosure, but the object is not limited to this, and may be an object other than a finger, such as a stylus pen, a ballpoint pen, a mechanical pencil, or a straw.

[0184] In the above embodiment, an example in which the size of the instruction three-dimensional area 38 is changed in conjunction with the change in the size of the observation three-dimensional area 36 has been described, but the technology of the present disclosure is not limited to this, and the size of the instruction three-dimensional area 38 may not be changed regardless of the change in the size of the observation three-dimensional area 36. In the example shown in FIG. 32, even if the overhead image is enlarged or reduced and displayed on the display 34A of the tablet terminal 34, the size of the observation three-dimensional area 36 and the size of the instruction three-dimensional area 38 may not be changed. Here, the display 34A is an example of the "first display device" and the "second display device" according to the technology of the present disclosure.

[0185] In addition, in the tablet terminal 34, in a state where a specific reference plane image 40B1 (see FIG. 33) showing an image capturing surface by the imaging device 18 with respect to the soccer field 24 is displayed on the display 34A, an instruction to enlarge or reduce the specific reference plane image 40B1 (hereinafter referred to as an "image size change instruction") is received by the touch panel 106A (an example of a "reception unit (reception device)" according to the technology of the present disclosure). When the specific reference plane image 40B1 is enlarged or reduced according to the image size change instruction received by the touch panel 106A, a three-dimensional area 36B in the real space corresponding to the specific reference plane image is generated by the change unit 58F in response to this. In the example shown in FIG. 33, a three-dimensional area 36B is shown in the case where the specific reference plane image 40B1 is enlarged by receiving an instruction to enlarge the specific reference plane image 40B1 as an image size change instruction by the touch panel 106A in a state where the specific reference plane image 40B1 is displayed on the display 34A. In the example shown in FIG. 33, the three-dimensional area 36B is located at the center of the observation three-dimensional area 36. An example of an instruction to enlarge the specific reference plane image 40B1 is a pinch-out operation on the touch panel 106A. In the example shown in FIG. 34, the three-dimensional area 36B is shown when the specific reference plane image 40B1 is displayed on the display 34A and an instruction to reduce the specific reference plane image 40B1 is received by the touch panel 106A as an image size change instruction, and the specific reference plane image 40B1 is reduced. In the example shown in FIG. 34, the three-dimensional area 36B is located outside the observation three-dimensional area 36 (the side surrounding the observation three-dimensional area 36). An example of an instruction to reduce the specific reference plane image 40B1 is a pinch-in operation on the touch panel 106A. Here, a pinch out operation and a pinch in operation are shown as examples of an image size change instruction, but the technology of the present disclosure is not limited to this. For example, an image size change instruction may be an operation of a specific soft key and / or a specific hard key, etc., as long as it is an operation defined as an image size change instruction.

[0186] When the size of the specific reference plane image 40B1 is changed according to the image size change instruction, the positional relationship information 60G (see FIG. 12) is updated by the change unit 58F. The positional relationship information 60G includes information indicating a relative positional relationship between the instruction three-dimensional area 38 and the three-dimensional area 36B in the real space corresponding to the specific reference plane image 40B1 enlarged or reduced according to the image size change instruction received by the touch panel 106A. When the image size change instruction is received by the touch panel 106A, the positional relationship information 60G is changed by the change unit 58F according to the received image size change instruction. The observation viewpoint and the observation line of sight are changed in the three-dimensional area 36B according to the instruction three-dimensional area 38 given by the spectator 28.

[0187] That is, the similarity relationship between the three-dimensional observation area 36 and the three-dimensional instruction area 38 is maintained, but the instruction given by the spectator 28 to the three-dimensional instruction area 38 is reflected in the three-dimensional area 36B (the observation viewpoint and observation line of sight are set within the three-dimensional area 36B). Note that, although an example in which the three-dimensional observation area 36 and the three-dimensional area 36B exist separately has been described here, the technology of the present disclosure is not limited to this, and the three-dimensional observation area 36 may be re-set as the three-dimensional area 36B.

[0188] In the example shown in Fig. 33, a part of the three-dimensional area in the observation three-dimensional area 36 is set as the three-dimensional area 36B corresponding to the instruction three-dimensional area 38 defined on the display 34A, and in the example shown in Fig. 34, a three-dimensional area outside the observation three-dimensional area 36 is set as the three-dimensional area 36B corresponding to the instruction three-dimensional area 38 defined on the display 34A, but the technology of the present disclosure is not limited thereto. For example, regardless of the enlargement or reduction of the specific reference plane image displayed on the display 34A, the observation viewpoint and the observation line of sight may be set in the observation three-dimensional area 36 according to the instruction three-dimensional area 38 given by the spectator 28, as in the above embodiment.

[0189] As an example, as shown in FIG. 35, by changing the position and orientation of a finger within a three-dimensional pointing area 38, the observation viewpoint and observation line of sight are changed in the same way as in the above embodiment.

[0190] In this way, the display surface of display 34A in a state where an overhead image is displayed on display 34A is set as instruction reference surface 40B, and CPU 118 generates instruction three-dimensional area 38 based on instruction reference surface 40B, thereby making it possible to generate instruction three-dimensional area 38 whose relationship with observation three-dimensional area 36 can be grasped. In addition, because an overhead image is displayed on display 34A, it is possible to generate instruction three-dimensional area 38 whose relationship with observation three-dimensional area 36 in an overhead state can be grasped.

[0191] In the above embodiment, the instruction reference plane 40A is generated based on the reference object image 41 obtained by capturing an image of the recording medium P by the imaging device 84 of the smartphone 14, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 36, the instruction reference plane 40C may be generated based on an image obtained by capturing an arbitrary area (the thighs of the spectator 28 in the example shown in FIG. 36) by the imaging device 84 of the smartphone 14. Also, the smartphone 14 may be sequentially placed at four locations, and the reception device 76 may receive an instruction for positioning at each location, thereby determining four points for generating the instruction reference plane 40C. The positional relationship between the four points is specified by using acceleration information obtained from the acceleration sensor 72, and the instruction reference plane 40C is uniquely determined from the specified positional relationship. When the instruction reference plane 40C is determined in this manner, the generation unit 88B generates an instruction three-dimensional area 38 based on the instruction reference plane 40C, as shown in FIG. 37 as an example.

[0192] In the above, four points are specified to define each of the pointing reference surfaces 40A, 40B, and 40C, but the technology of the present disclosure is not limited to this, and the pointing reference surfaces 40A, 40B, and / or 40C may be defined by three points, or any number of points that can define a surface is sufficient. In the following, when there is no need to distinguish between the pointing reference surfaces 40A, 40B, and 40C, they will be referred to as "pointing reference surfaces" without reference numerals.

[0193] When identifying at least three or more points required to generate a reference surface for pointing, for example, as shown in FIG. 38, a finger may be placed in any four locations in sequence, and multiple points may be identified from an image obtained by capturing an image of the finger at each location using the imaging device 84 of the smartphone 14.

[0194] As an example, as shown in FIG. 39, the smartphone 14 may be rotated around the roll axis, and the virtual viewpoint image 46 may be generated by the virtual viewpoint image generating unit 58C2 in a direction corresponding to the smartphone attitude detected by measuring the roll angle by the detection unit 88A. This makes it easier to obtain the virtual viewpoint image 46 in a direction corresponding to the roll angle compared to a case where the roll angle is not measured. Note that the example shown in FIG. 39 shows a mode in which, when the smartphone 14 is rotated 90 degrees around the roll axis, the virtual viewpoint image 46 is also rotated 90 degrees and generated by the virtual viewpoint image generating unit 58C2.

[0195] In the above embodiment, the positional relationship information 60G in which the observation three-dimensional region information 60F and the instruction three-dimensional region information 90C are associated with each other in a table format is illustrated, but the technology of the present disclosure is not limited to this. For example, the positional relationship information 60G may be information including the difference between the distance between the position in the observation three-dimensional region 36 and the observation reference point 36A and the distance between the position in the instruction three-dimensional region 38 and the instruction reference point 38A. This makes it easier to specify the relationship between the position in the observation three-dimensional region 36 and the position in the instruction three-dimensional region 38, compared to when the difference between the distance between the position in the observation three-dimensional region 36 and the point unrelated to the observation reference point 36A and the distance between the position in the instruction three-dimensional region 38 and the point unrelated to the instruction reference point 38A is used as the positional relationship information.

[0196] An example of the degree of difference is the ratio of the distance between a position in the observation three-dimensional area 36 and the observation reference point 36A to the distance between a position in the instruction three-dimensional area 38 and the instruction reference point 38A, that is, the magnification. Instead of the magnification, a difference may be used as the degree of difference. Examples of information including the degree of difference include, The degree of difference, Information indicating the direction of a position within the three-dimensional area 36 for observation relative to the reference point 36A for observation and, Information indicating the direction of a position in the three-dimensional area 38 for indication relative to the reference point 38A for indication and information including Examples include:

[0197] In the above embodiment, the observation reference point 36A and the instruction reference point 38A are exemplified, but the technology of the present disclosure is not limited thereto, and a single reference point common to the observation three-dimensional area 36 and the instruction three-dimensional area 38 (hereinafter also referred to as a "common reference point") may be adopted. For example, the center of the center circle of the soccer field 24 or one of the four corners of the soccer field 24 may be used as the common reference point. In this case, the distance between the position in the observation three-dimensional area 36 and the common reference point, and the distance between the position in the instruction three-dimensional area 38 and the common reference point to the other, that is, information including a magnification, may be adopted as the positional relationship information 60G.

[0198] In the above embodiment, the indication reference surface 40A is an outer surface of the indication three-dimensional area 38, but the technology of the present disclosure is not limited to this. The indication reference surface 40A may be a surface that defines one of the outer surfaces of the indication reference surface 40A (for example, a surface that is included in one of the outer surfaces of the indication reference surface 40A), or may be an internal surface of the indication three-dimensional area 38.

[0199] In the above embodiment, the image generating device 12 is provided with the acquisition unit 58C. However, the technology of the present disclosure is not limited to this. The acquisition unit 58C may be provided in an external device of the image generating device 12. Here, the external device may be, for example, a smartphone 14 or a tablet terminal 34. Alternatively, the acquisition unit 58C may be provided in a server or a personal computer. In this case, the viewpoint line of sight information derived by the derivation unit 58B is transmitted from the image generating device 12 to the acquisition unit 58C of the external device, and the viewpoint line of sight information acquisition unit 58C1 of the acquisition unit 58C acquires the viewpoint line of sight information. The external device may acquire the viewpoint line of sight information, generate a virtual viewpoint image based on the acquired viewpoint line of sight information, and the image generating device 12 may acquire the virtual viewpoint image generated by the external device.

[0200] In the above embodiment, the instruction three-dimensional area 38 is a three-dimensional area obtained by reducing the observation three-dimensional area 36, ​​but the technology of the present disclosure is not limited to this. For example, the instruction three-dimensional area 38 may be a three-dimensional area obtained by enlarging the observation three-dimensional area 36. In this case, an application example in a medical field is considered, such as applying the observation three-dimensional area 36 to a surgical field for various surgeries such as ophthalmology and / or neurosurgery, and applying the instruction three-dimensional area 38 to the field of view of the surgeon. In addition to the medical field, in a scene where a minute object such as a cell is observed, the observation three-dimensional area 36 may be applied to the observation target, and the instruction three-dimensional area 38 may be applied to the field of view of the observer.

[0201] In the above embodiment, the area matching process (see FIG. 27), the virtual viewpoint image generation process (see FIG. 28), the image control process (see FIG. 29), the change process (see FIG. 30), and the change speed instruction process (see FIG. 31) are executed by the CPU 58 of the image generating device 12. However, the technology of the present disclosure is not limited to this. For example, at least one of the area matching process, the virtual viewpoint image generation process, the image control process, the change process, and the change speed instruction process may be executed by the CPU 88 of the smartphone 14.

[0202] 40 shows a configuration example of the smartphone 14 in the case where the area association process, the virtual viewpoint image generation process, the image control process, the change process, and the change speed instruction process are executed by the CPU 88 of the smartphone 14. As shown in FIG. 40 as an example, the storage 90 stores an area association program 60A, a virtual viewpoint image generation program 60B, an image control program 60C, a change program 60D, and a change speed instruction program 60E in addition to the generation program 90A and the detection program 90B. The CPU 88 operates as the area association unit 58A, the derivation unit 58B, the acquisition unit 58C, the output unit 58D, the image control unit 58E, the change unit 58F, and the change speed instruction unit 58G in addition to operating as the detection units 88A and 88B. That is, the CPU 88 operates as an area matching unit 58A, a derivation unit 58B, an acquisition unit 58C, an output unit 58D, an image control unit 58E, a change unit 58F, and a change rate instruction unit 58G in accordance with the image generating device side program stored in the storage 90, thereby executing area matching processing, virtual viewpoint image generation processing, image control processing, change processing, and change rate instruction processing.

[0203] In the example shown in Fig. 40, the smartphone 14 is an example of an "information processing device" according to the technology of the present disclosure. In the example shown in Fig. 40, the smartphone 14 is given as an example of an "information processing device" according to the technology of the present disclosure, but instead of the smartphone 14, a tablet terminal 34 can also be adopted as the "information processing device" according to the technology of the present disclosure. In addition, various devices with a computing device, such as a personal computer and / or a wearable terminal, can also be adopted as the "information processing device" according to the technology of the present disclosure.

[0204] In addition, in the above embodiment, a soccer stadium 22 is used as an example, but this is merely one example and the venue can be any venue such as a baseball stadium, curling court, swimming stadium, etc., as long as multiple imaging devices can be installed and there is equipment capable of generating virtual viewpoint images.

[0205] In addition, in the above embodiment, a wireless communication system using the base station 20 is exemplified, but this is merely one example, and the technology of the present disclosure is also applicable to a wired communication system using a cable.

[0206] In addition, while the above embodiment illustrates an unmanned aerial vehicle 27, the technology of the present disclosure is not limited to this, and the three-dimensional observation area 36 may be imaged by an imaging device 18 suspended by a wire (for example, a self-propelled imaging device that can move along a wire).

[0207] In addition, in the above embodiment, an example was given of a case in which spectators 28 (operators of devices such as smartphone 14) are watching soccer at an actual soccer stadium 22, but this is not limited to the above case. The technology disclosed herein can also be applied to a case in which spectators 28 are not at the soccer stadium 22 but are watching soccer on a television at home or the like.

[0208] In the above embodiment, an example in which the imaging device 18 is used has been described, but the technology of the present disclosure is not limited to this. For example, when the relative positional relationship between the observation three-dimensional area 36 and the instruction three-dimensional area 38 is associated with each other and an overhead image is not displayed, the technology of the present disclosure can be realized even without the imaging device 18.

[0209] In addition, in the above embodiment, the computers 50, 70, and 100 are exemplified, but the technology of the present disclosure is not limited thereto. For example, devices including ASIC, FPGA, and / or PLD may be applied instead of the computers 50, 70, and / or 100. Also, a combination of hardware configuration and software configuration may be used instead of the computers 50, 70, and / or 100.

[0210] Furthermore, in the above embodiment, the image generating device-side program is stored in the storage 60, but the technology of the present disclosure is not limited to this, and as an example, the image generating device-side program may be stored in any portable storage medium 200 such as an SSD or USB memory, which is a non-transitory storage medium, as shown in Fig. 41. In this case, the image generating device-side program stored in the storage medium 200 is installed in the computer 50, and the CPU 58 executes the area association process, the virtual viewpoint image generation process, the image control process, the change process, and the change speed instruction process in accordance with the image generating device-side program.

[0211] Also, the image generating device-side program may be stored in a storage unit of another computer or a server device or the like connected to computer 50 via a communication network (not shown), and the image generating device-side program may be downloaded to image generating device 12 in response to a request from image generating device 12. In this case, the area matching process, virtual viewpoint image generating process, image control process, change process, and change speed instruction process based on the downloaded image generating device-side program are executed by CPU 58 of computer 50.

[0212] In addition, in the above embodiment, the CPU 58 is exemplified, but the technology of the present disclosure is not limited to this, and a GPU may be adopted. Moreover, multiple CPUs may be adopted instead of the CPU 58. That is, the area matching process, the virtual viewpoint image generating process, the image control process, the change process, and the change speed instruction process may be executed by one processor or multiple processors that are physically separated from each other.

[0213] In the above embodiment, the smartphone-side program is stored in the storage 90, but the technology of the present disclosure is not limited to this, and as an example, the smartphone-side program may be stored in any portable storage medium 300 such as an SSD or a USB memory, as shown in Fig. 42. In this case, the smartphone-side program stored in the storage medium 300 is installed in the computer 70, and the CPU 88 executes the detection process (see Fig. 26) and the generation process (see Fig. 25) in accordance with the smartphone-side program.

[0214] Furthermore, the smartphone-side program may be stored in a storage unit of another computer or a server device connected to the computer 70 via a communication network (not shown), and the smartphone-side program may be downloaded to the smartphone 14 in response to a request from the smartphone 14. In this case, the detection process and generation process based on the downloaded smartphone-side program are executed by the CPU 88 of the computer 70.

[0215] In addition, in the above embodiment, the CPU 88 is exemplified, but the technology of the present disclosure is not limited to this, and a GPU may be adopted. Moreover, multiple CPUs may be adopted instead of the CPU 88. That is, the detection process and the generation process may be executed by one processor, or multiple processors that are physically separated. In the following, for convenience of explanation, the area matching process, the virtual viewpoint image generation process, the image control process, the change process, the change speed instruction process, the detection process, and the generation process are referred to as "various processes" when there is no need to distinguish between them.

[0216] As hardware resources for executing various processes, the following various processors can be used. As described above, an example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes according to software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGA, PLD, or ASIC, which are processors having a circuit configuration designed specifically for executing specific processes. All of the processors have built-in or connected memory, and all of the processors use the memory to execute various processes.

[0217] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0218] As an example of a configuration using one processor, first, there is a configuration in which one processor is configured by combining one or more CPUs and software, as typified by computers such as client and server, and this processor functions as a hardware resource that executes various processes. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute various processes in a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0219] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0220] In addition, the various processes described above are merely examples, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0221] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0222] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0223] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0224] The following supplementary notes are further disclosed regarding the above embodiment.

[0225] (Appendix 1) A processor; A memory included in or connected to the processor; The processor is Detecting a three-dimensional position and orientation of an object within a three-dimensional pointing region having a scaled or reduced relationship to a three-dimensional observation region defining a virtual viewpoint and line of sight; deriving the viewpoint and the line of sight corresponding to the detection result according to positional relationship information indicating a relative positional relationship between the three-dimensional observation area and the three-dimensional instruction area; A virtual viewpoint image is obtained based on a plurality of images obtained by capturing images of an imaging region included in the three-dimensional region for observation by a plurality of imaging devices, the virtual viewpoint image showing the subject when observed from the derived viewpoint and line of sight. Information processing device.

[0226] (Appendix 2) a detection unit that detects a three-dimensional position and orientation of an object within a three-dimensional instruction area having an enlarged or reduced relationship with a three-dimensional observation area that defines a virtual viewpoint and line of sight; a derivation unit that derives the viewpoint and the line of sight corresponding to the detection result of the detection unit according to positional relationship information indicating a relative positional relationship between the three-dimensional observation area and the three-dimensional instruction area; an acquisition unit that acquires a virtual viewpoint image based on a plurality of images obtained by capturing images of an imaging region included in the three-dimensional observation region by a plurality of imaging devices, the virtual viewpoint image showing the subject when the subject is observed from the viewpoint and the line of sight derived by the derivation unit; An information processing device comprising:

[0227] (Appendix 3) a generation unit that generates the instruction three-dimensional area based on an instruction reference surface identified by using an image obtained by capturing an image of the reference object; The information processing device according to claim 2, wherein the reference subject is a recording medium on which an image showing a specific subject (in the example shown in FIG. 1, a soccer field 24) within the three-dimensional observation area is formed.

[0228] According to the information processing device described in Supplementary Note 3, it is possible to generate an instruction three-dimensional area that allows one to grasp a relationship with a specific subject in the observation three-dimensional area.

[0229] (Appendix 4) The three-dimensional observation area is a three-dimensional area defined based on a second observation three-dimensional area image obtained by imaging an area including a specific reference plane using a second imaging device or obtained by imaging an area including a specific reference plane using a second virtual imaging device, and is in a similar relationship to the three-dimensional instruction area; a reception unit for receiving a size change instruction for changing the size of the three-dimensional observation area; an information processing device according to claim 2 or 3, comprising: a change unit that changes the size of the observation three-dimensional area by changing the width of an imaging range relative to the specific reference plane by the second imaging device or the second virtual imaging device in accordance with the size change instruction received by the receiving unit, and changes the size of the instruction three-dimensional area while maintaining the similarity relationship in conjunction with the change in the size of the observation three-dimensional area.

[0230] According to the information processing device described in Supplementary Note 4, a change in the size of the three-dimensional observation region can be reflected in the three-dimensional instruction region.

[0231] (Appendix 5) the receiving unit receives an image size change instruction to enlarge or reduce a specific reference plane image in a state in which an image including a specific reference plane image indicating the specific reference plane is displayed by a second display device, The information processing device described in Appendix 4, wherein the positional relationship information includes information indicating the relative positional relationship between a three-dimensional area in real space corresponding to the specific reference plane image enlarged or reduced in accordance with the image size change instruction received by the receiving unit and the instruction three-dimensional area.

[0232] According to the information processing device described in Supplementary Note 5, the position of the viewpoint can be determined without being restricted by the three-dimensional area for observation.

[0233] (Appendix 6) An information processing device as described in Appendix 4 or Appendix 5, wherein the size of the virtual viewpoint image acquired by the acquisition unit changes at a speed determined according to the degree of change in size of the three-dimensional area for observation and the three-dimensional area for instruction.

[0234] According to the information processing device described in Appendix 6, it is possible to reduce visual discomfort when the size of the virtual viewpoint image is changed, compared to when the size of the virtual viewpoint image is changed without taking into account the degree of change in size of the three-dimensional observation area and the three-dimensional instruction area.

Claims

1. A processor; A memory included in or connected to the processor; The processor, receiving input of a virtual viewpoint and line of sight within a three-dimensional instruction area having a scaled or reduced relationship to a three-dimensional observation area defining the viewpoint and line of sight; A virtual viewpoint image when the observation three-dimensional area is observed from the viewpoint and the line of sight is obtained. Information processing device.

2. The information processing apparatus according to claim 1 , wherein the relative positional relationship between the three-dimensional observation area and the three-dimensional instruction area is a relationship between a position of the three-dimensional observation area with respect to a reference point and a position of the three-dimensional instruction area with respect to the reference point.

3. 3. The information processing device according to claim 2, wherein the positional relationship information indicating the relative positional relationship is information in which coordinates indicating a position within the three-dimensional observation area with the reference point as the origin correspond to coordinates indicating a position within the three-dimensional instruction area with the reference point as the origin.

4. When the reference points are classified into observation reference points applied to the observation three-dimensional area and indication reference points applied to the indication three-dimensional area, the indication reference point is in a corresponding relationship with the observation reference point at a position different from the observation reference point, the positional relationship information indicating the relative positional relationship is information including a degree of difference between a distance between a position in the observation three-dimensional area and the observation reference point and a distance between a position in the indication three-dimensional area and the indication reference point, When the reference point is a single reference point common to the three-dimensional observation area and the three-dimensional indication area, The positional relationship information indicating the relative positional relationship is information including a degree of difference between a distance between a position in the three-dimensional observation area and the single reference point, and a distance between a position in the three-dimensional indication area and the single reference point. The information processing device according to claim 2 .

5. The information processing apparatus according to claim 4 , wherein the indication reference point is determined by detecting a three-dimensional position of an object within the indication three-dimensional area by the processor.

6. The processor, Detecting three-dimensional positions of an object within the three-dimensional indication region of at least three points; The information processing apparatus according to claim 1 , wherein the instruction three-dimensional area is generated based on an instruction reference plane that is specified by using the three-dimensional positions of the at least three detected points.

7. The information processing device according to claim 1 , wherein the processor generates the three-dimensional instruction area based on an instruction reference surface that is identified by using an image obtained by capturing an image of a reference subject.

8. a display surface of a first three-dimensional area image for observation obtained by imaging the three-dimensional area for observation with a first imaging device or obtained by imaging the three-dimensional area for observation with a first virtual imaging device is displayed on a first display device, the display surface of the first three-dimensional area image for observation being used as a reference surface for indication; The information processing device according to claim 1 , wherein the processor generates the instruction three-dimensional area based on the instruction reference surface.

9. The information processing apparatus according to claim 8 , wherein the first observation three-dimensional area image is an image showing the observation three-dimensional area in a bird's-eye view.

10. 10. The information processing apparatus according to claim 8, wherein the processor enlarges or reduces the first observation three-dimensional area image in accordance with a given instruction.

11. The information processing device according to claim 6 , wherein the pointing reference surface is one bottom surface of the pointing three-dimensional area.

12. The processor detects an orientation of an object within the three-dimensional area for instruction by measuring a pitch angle, a yaw angle, and a roll angle of the object within the three-dimensional area for instruction; The information processing apparatus according to claim 1 , wherein the virtual viewpoint image is acquired in a direction corresponding to the attitude detected by measuring the roll angle.

13. a receiving device that receives an image size change instruction for enlarging or reducing a specific reference plane image in a state in which an image including the specific reference plane image showing the specific reference plane is displayed by a second display device, An information processing device as described in any one of claims 1 to 12, wherein the positional relationship information indicating the relative positional relationship between the observation three-dimensional area and the instruction three-dimensional area includes information indicating the relative positional relationship between the instruction three-dimensional area and a three-dimensional area in real space corresponding to the specific reference plane image enlarged or reduced in accordance with the image size change instruction accepted by the accepting device.

14. The information processing device according to claim 1 , wherein the processor outputs the acquired virtual viewpoint image.

15. receiving input of a virtual viewpoint and line of sight within a three-dimensional pointing area having a scaled or reduced relationship to a three-dimensional observation area defining the viewpoint and line of sight; and and acquiring a virtual viewpoint image when the three-dimensional observation area is observed from the viewpoint and the line of sight. Information processing methods.

16. On the computer, receiving input of a virtual viewpoint and line of sight within a three-dimensional pointing area having a scaled or reduced relationship to a three-dimensional observation area defining the viewpoint and line of sight; and A program for executing a process including acquiring a virtual viewpoint image when the three-dimensional observation area is observed from the viewpoint and the line of sight.

Images