Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program which enable a user to observe a state inside a three-dimensional region from various positions.SOLUTION: An information processing apparatus includes a processor, and a memory built in or connected to the processor. The processor acquires a subject image showing a subject present inside a three-dimensional region when the three-dimensional region is observed from a viewpoint position determined based on coordinates inside the three-dimensional region corresponding to an indication position that is indicated inside the three-dimensional region to be observed or that is indicated inside a reference image showing a state of the inside of the three-dimensional region in a case in which the inside of the three-dimensional region is observed from a reference position.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] JP 2019-133309 A discloses a program for causing a computer to execute the steps of: setting a virtual space for providing a user with a virtual experience; setting a plurality of movement areas in the virtual space; setting a virtual viewpoint in the virtual space; indicating a predetermined movement area among the plurality of movement areas in accordance with movement of a part of the user's body; moving the virtual viewpoint to the predetermined movement area if the distance between the virtual viewpoint and the predetermined movement area is equal to or less than a first threshold; and not moving the virtual viewpoint to the predetermined movement area if the distance between the virtual viewpoint and the predetermined movement area exceeds the first threshold. Summary of the Invention

[0003] One embodiment of the technology disclosed herein provides an information processing device, an information processing method, and a program that enable a user to observe a state within a three-dimensional area from various positions. [Means for solving the problem]

[0004] A first aspect of the technology disclosed herein is an information processing device that includes a processor and a memory built into or connected to the processor, and acquires a subject image that shows a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position that is determined based on coordinates within the three-dimensional area that correspond to an indicated position within the three-dimensional area that is being observed, or an indicated position within a reference image that shows the appearance of the three-dimensional area when the three-dimensional area is observed from a reference position.

[0005] A second aspect of the technology of the present disclosure is an information processing device according to the first aspect, in which the processor derives coordinates based on an observation mode in which the three-dimensional area is observed and a pointed position.

[0006] A third aspect according to the technology of the present disclosure is the information processing device according to the second aspect, in which the observation mode is determined according to an observation position at which the three-dimensional region is observed.

[0007] A fourth aspect according to the technology of the present disclosure is an information processing device according to the third aspect, in which a processor determines an observation position designation range in which an observation position can be designated according to an attribute of a designation source.

[0008] A fifth aspect of the technology of the present disclosure is an information processing device relating to the fourth aspect, in which a processor acquires an image of the appearance within the three-dimensional area showing the appearance within the three-dimensional area when the three-dimensional area is observed in an observation manner, and the image of the appearance within the three-dimensional area is an image shown in a manner in which the observation position indication range within the three-dimensional area and the range outside the observation position indication range are distinguishable.

[0009] A sixth aspect of the technology of the present disclosure is the information processing device according to the fifth aspect, in which the reference image is an image based on the three-dimensional area interior aspect image.

[0010] A seventh aspect of the technology of the present disclosure is an information processing device relating to any one of the second to sixth aspects, in which a processor derives coordinates based on a correspondence between an image showing the appearance of the three-dimensional area when the three-dimensional area is observed in an observation mode and a three-dimensional area image showing the three-dimensional area and whose position can be identified by coordinates.

[0011] An eighth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to seventh aspects, in which the reference image is a virtual viewpoint image generated based on multiple images obtained by capturing images of a three-dimensional area using multiple imaging devices, or an image based on captured images obtained by capturing images of a three-dimensional area.

[0012] A ninth aspect according to the technology of the present disclosure is the information processing device according to the eighth aspect, in which the designated position designated in the reference image is a specific position in the virtual viewpoint image or the captured image.

[0013] A tenth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to ninth aspects, wherein the reference image is an image including a first mark that can identify an indication position within the reference image.

[0014] An eleventh aspect of the technology disclosed herein is an information processing device relating to any one of the first to tenth aspects, in which the subject image includes a second mark capable of identifying a designated position designated within the reference image.

[0015] A twelfth aspect of the technology disclosed herein is an information processing device relating to any one of the first to eleventh aspects, in which a processor acquires an object image instead of a subject image when an object image showing an object present within a three-dimensional region is stored in the storage area when the three-dimensional region is observed from a position whose distance from the indicated position is within a threshold range.

[0016] A thirteenth aspect of the technology disclosed herein is an information processing device according to any one of the first to twelfth aspects, wherein coordinates for a specific area within a three-dimensional area indicate a position higher than the actual position of the specific area within the three-dimensional area.

[0017] A fourteenth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to thirteenth aspects, wherein the designated position designated within the three-dimensional region is a designated position on a first line extending from a viewpoint at which the three-dimensional region is observed to a point of gaze, and the designated position designated within the reference image is a designated position on a second line extending from the reference position to a specified point in the reference image.

[0018] A fifteenth aspect of the technology disclosed herein is an information processing device relating to any one of the first to fourteenth aspects, wherein a designated position designated within the three-dimensional region is a position selected from at least one first candidate position, and a designated position designated within the reference image is a position selected from at least one second candidate position, and the processor associates a first reduced-size image obtained by reducing the image of the subject when the three-dimensional region is observed from the first candidate position with the at least one first candidate position, and associates a second reduced-size image obtained by reducing the image of the subject when the three-dimensional region is observed from the second candidate position with the at least one second candidate position.

[0019] A 16th aspect of the technology disclosed herein is an information processing device relating to any one of the first to fifteenth aspects, in which a processor detects a pointed position based on a designated area image showing an area specified within a three-dimensional area.

[0020] A 17th aspect of the technology of the present disclosure is an information processing device relating to any one of the first to sixteenth aspects, in which the subject image is a virtual viewpoint image generated based on multiple images obtained by capturing images of a three-dimensional area by multiple imaging devices.

[0021] An 18th aspect of the technology of the present disclosure is an information processing method that includes acquiring a subject image that shows a subject present in a three-dimensional area when the three-dimensional area is observed from a viewpoint position that is determined based on coordinates within the three-dimensional area that correspond to an indicated position within the three-dimensional area that is the object of observation, or an indicated position within a reference image that shows the appearance of the three-dimensional area when the three-dimensional area is observed from a reference position.

[0022] A 19th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a subject image showing a subject present in a three-dimensional area when observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indicated position within the three-dimensional area being observed, or indicated within a reference image showing the appearance of the three-dimensional area when observed from a reference position. [Brief description of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of a configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a conceptual diagram showing an example of a configuration of a three-dimensional area image. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of a user device. FIG. [Figure 4] FIG. 1 is a schematic perspective view showing an example of an aspect in which the inside of a soccer stadium is captured by an imaging device of a smart device. [Diagram 5] FIG. 4 is a conceptual diagram showing an example of the contents of a user device side process according to the first embodiment. [Figure 6] FIG. 4 is a conceptual diagram showing an example of the contents of a user device side process according to the first embodiment. [Figure 7] FIG. 4 is a conceptual diagram showing an example of the contents of a user device side process according to the first embodiment. [Figure 8] FIG. 4 is a conceptual diagram showing an example of the contents of a user device side process according to the first embodiment. [Figure 9] FIG. 4 is a conceptual diagram showing an example of the contents of an image generating process according to the first embodiment. [Figure 10] FIG. 4 is a conceptual diagram showing an example of the contents of an image generating process according to the first embodiment. [Figure 11] 10 is a flowchart showing an example of the flow of a user device side process according to the first embodiment. [Figure 12] 5 is a flowchart showing an example of the flow of image generation processing according to the first embodiment. [Figure 13] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the first embodiment. [Figure 14] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the first embodiment. [Figure 15] FIG. 2 is a conceptual diagram showing an example of the configuration of an HMD. [Figure 16] FIG. 11 is a conceptual diagram showing an example of the contents of HMD-side processing according to the second embodiment. [Figure 17] FIG. 2 is a conceptual diagram showing an example of a display mode of a display of an HMD. [Figure 18] 11A and 11B are conceptual diagrams used to explain a method for setting a tentative designated position. [Figure 19] FIG. 11 is a conceptual diagram showing an example of the contents of HMD-side processing according to the second embodiment. [Figure 20] FIG. 13 is a conceptual diagram showing an example of the configuration of an HMD image with a tentative designated position. [Figure 21] FIG. 11 is a conceptual diagram showing an example of the contents of an image generating process according to the second embodiment. [Figure 22] 11 is a conceptual diagram showing an example of a configuration of a different viewpoint position image with a designated position candidate. FIG. [Diagram 23] FIG. 11 is a conceptual diagram showing an example of the contents of HMD-side processing according to the second embodiment. [Figure 24] FIG. 2 is a conceptual diagram showing an example of the display content of an HMD display. [Diagram 25] FIG. 11 is a conceptual diagram showing an example of the contents of HMD-side processing according to the second embodiment. [Figure 26] FIG. 11 is a conceptual diagram showing an example of the contents of an image generating process according to the second embodiment. [Figure 27] 13 is a flowchart showing an example of the flow of a user device side process according to the second embodiment. [Figure 28] 10 is a flowchart showing an example of a flow of HMD-side processing according to the second embodiment. [Figure 29] This is a continuation of the flowchart shown in Figure 28. [Diagram 30]10 is a flowchart showing an example of the flow of an image generation process according to the second embodiment. [Diagram 31] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the second embodiment. [Diagram 32] 1 is a schematic perspective view showing an example of an aspect in which images of a user's fingers are captured by a plurality of imaging devices; [Diagram 33] 11 is a conceptual diagram showing an example of a manner in which a different viewpoint position image with a pointing position candidate is displayed on the display of a user device. [Diagram 34] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the second embodiment. [Diagram 35] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the second embodiment. [Diagram 36] 11 is a conceptual diagram showing an example of a configuration of a different viewpoint position image with a designated position candidate. FIG. [Figure 37] FIG. 11 is a conceptual diagram showing a modified example of the content of the HMD-side processing according to the second embodiment. [Figure 38] FIG. 11 is a conceptual diagram showing a modified example of the content of the image generating process according to the second embodiment. [Figure 39] 11 is a conceptual diagram showing an example of a manner in which a different viewpoint position image with a pointing position candidate is displayed on the display of a user device. [Diagram 40] FIG. 13 is a conceptual diagram illustrating an example of a configuration of an information processing system according to a third embodiment. [Diagram 41] FIG. 13 is a conceptual diagram showing an example of the configuration of a three-dimensional area image with spectator seating information. [Diagram 42] FIG. 13 is a conceptual diagram showing an example of the contents of a user device side process according to the third embodiment. [Diagram 43] FIG. 13 is a conceptual diagram showing an example of the contents of an image generating process according to the third embodiment. [Diagram 44] FIG. 13 is a conceptual diagram showing an example of the contents of a user device side process according to the third embodiment. [Diagram 45] FIG. 13 is a conceptual diagram showing an example of a reference image being displayed on a display of a user device. [Figure 46]13 is a flowchart showing an example of the flow of a user device side process according to the third embodiment. [Figure 47] 13 is a flowchart showing an example of the flow of observation range restriction processing according to the third embodiment. [Figure 48] FIG. 2 is a conceptual diagram showing an example of a manner in which a program stored in a storage medium is installed in a computer of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] An example of an embodiment of an information processing device, an information processing method, and a program according to the technology of the present disclosure will be described with reference to the attached drawings.

[0025] First, the terms used in the following description will be explained.

[0026] CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-Volatile Memory". RAM is an abbreviation for "Random Access Memory". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". GPU is an abbreviation for "Graphics Processing Unit". LAN is an abbreviation for "Local Area Network". 3D is an abbreviation for "3 Dimensions". USB is an abbreviation for "Universal Serial Bus". "HMD" is an abbreviation for "Head Mounted Display". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)". TDM is an abbreviation for "Time-Division Multiplexing". HMD is an abbreviation for "Head Mounted Display".For ease of explanation, a CPU is exemplified below as an example of a "processor" according to the technology of the present disclosure, but the "processor" according to the technology of the present disclosure may be a combination of multiple processing devices such as a CPU and a GPU. When a combination of a CPU and a GPU is applied as an example of a "processor" according to the technology of the present disclosure, the GPU operates under the control of the CPU and is responsible for executing image processing.

[0027] In the following description, "match" refers not only to a perfect match, but also to a match that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not go against the spirit of the technology of the present disclosure.

[0028] [First embodiment] As an example, as shown in FIG. 1, an information processing system 2 includes an information processing device 10 and a user device 12.

[0029] In the first embodiment, a server is applied as an example of the information processing device 10. However, this is merely an example, and the information processing device 10 may be a personal computer, a plurality of personal computers, a plurality of servers, or a device that combines a personal computer and a server.

[0030] In the first embodiment, a smartphone is applied as an example of the user device 12. However, the smartphone is merely an example, and the user device 12 may be, for example, a personal computer, a tablet terminal, an HMD, or other portable multi-function terminal.

[0031] In the first embodiment, the information processing device 10 and the user device 12 are communicatively connected via, for example, a base station (not shown). The communication standards used in the base station include wireless communication standards including the 5G standard, the LTE standard, etc., wireless communication standards including the WiFi (802.11) standard and / or the Bluetooth (registered trademark) standard, and wired communication standards including the TDM standard and / or the Ethernet (registered trademark) standard.

[0032] The information processing device 10 acquires an image and transmits the acquired image to the user device 12. Here, the image refers to, for example, a captured image obtained by capturing an image, an image generated based on the captured image, etc. An example of an image generated based on a captured image is a virtual viewpoint image.

[0033] The user device 12 is used by a user 13. The user device 12 includes a touch panel display 16. The touch panel display 16 is realized by a display 18 and a touch panel 20. An example of the display 18 is an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0034] The touch panel display 16 is formed by overlapping a touch panel 20 on the display area of ​​the display 18, or by forming an in-cell type in which a touch panel function is built into the inside of the display 18. Note that the in-cell type is merely one example, and the display may be an out-cell type or an on-cell type.

[0035] The user device 12 executes a process (e.g., a user device process described later) according to an instruction received from a user via the touch panel 20, etc. For example, the user device 12 transmits and receives various information to and from the information processing device 10 according to an instruction received from a user via the touch panel 20, etc.

[0036] The user device 12 receives the image transmitted from the information processing device 10, and displays the received image on the display 18. The user 13 views the image displayed on the display 18.

[0037] The information processing device 10 includes a computer 22, a transmission / reception device 24, a communication I / F 26, and a bus 28. The computer 22 includes a CPU 22A, an NVM 22B, and a RAM 22C, and the CPU 22A, the NVM 22B, and the RAM 22C are connected via the bus 28. In the example shown in Fig. 1, for convenience of illustration, one bus is shown as the bus 28, but multiple buses may be used. In addition, the bus 28 may include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, and the like.

[0038] The CPU 22A is an example of a "processor" according to the technology of the present disclosure. The CPU 22A controls the entire information processing device 10. The NVM 22B stores various parameters and various programs. Examples of the NVM 22B include an EEPROM, an SSD, and / or an HDD. The RAM 22C is an example of a "memory" according to the technology of the present disclosure. Various information is temporarily stored in the RAM 22C. The RAM 22C is used as a work memory by the CPU 22A.

[0039] The transmission / reception device 24 is connected to the bus 28. The transmission / reception device 24 is a device including a communication processor (not shown) and an antenna, and transmits and receives various types of information to and from the user device 12 via a base station (not shown) under the control of the CPU 22A. That is, the CPU 22A transmits and receives various types of information to and from the user device 12 via the transmission / reception device 24.

[0040] The communication I / F 26 is realized by, for example, a device having an FPGA. The communication I / F 26 is connected to a plurality of imaging devices 30 via a LAN cable (not shown). The imaging device 30 is an imaging device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor.

[0041] The multiple imaging devices 30 are installed in a soccer stadium 36 (see FIG. 2) and capture images of subjects within the soccer stadium 36. Captured images obtained by capturing images of subjects by the imaging devices 30 are used, for example, to generate a virtual viewpoint image. For this reason, the multiple imaging devices 30 are installed at different locations within the soccer stadium 36 (see FIG. 2), i.e., at locations where multiple captured images from which a virtual viewpoint image can be generated can be obtained.

[0042] The communication I / F 26 is connected to the bus 28, and controls the exchange of various information between the CPU 22A and the multiple imaging devices 30. For example, the communication I / F 26 controls the multiple imaging devices 30 in accordance with a request from the CPU 22A. The communication I / F 26 outputs captured images (hereinafter also simply referred to as "captured images") obtained by capturing images using each of the multiple imaging devices 30 to the CPU 22A. Note that, although the communication I / F 26 is exemplified here as a wired communication I / F, it may be a wireless communication I / F such as a high-speed wireless LAN.

[0043] The NVM 22B stores a three-dimensional area image 32 and an image generation processing program 34. Although details will be described later, the three-dimensional area image 32 is a three-dimensional image showing the state of a three-dimensional area, and the three-dimensional area image 32 is assigned coordinates that can specify a position within the three-dimensional area.

[0044] The image generation processing program 34 is an example of a “program” according to the technology of the present disclosure. The CPU 22A reads the image generation processing program 34 from the NVM 22B and executes the image generation processing program 34 on the RAM 22C to perform image generation processing (see FIG. 12).

[0045] As an example, as shown in FIG. 2, the three-dimensional area image 32 is a three-dimensional image showing a soccer stadium 36. The soccer stadium 36 is an example of a "three-dimensional area" according to the technology of the present disclosure. The soccer stadium 36 is a three-dimensional area including a soccer field 36A and spectator seats 36B constructed to surround the soccer field 36A, and is observed by a user 13. The example shown in FIG. 2 shows a state in which the observer, i.e., the user 13, is observing the inside of the soccer stadium 36 from the spectator seats 36B.

[0046] The three-dimensional area image 32 is provided with coordinates capable of identifying a position within the soccer stadium 36. Here, as an example of coordinates capable of identifying a position within the soccer stadium 36, three-dimensional coordinates capable of identifying a position within a rectangular parallelepiped 38 that surrounds the soccer stadium 36, with one vertex of the rectangular parallelepiped 38 as the origin, are applied.

[0047] The coordinates relating to the position of the soccer field 36A shown in the three-dimensional area image 32 are coordinates indicating a position higher than the actual position of the soccer field 36A. Here, the coordinates relating to the position of the soccer field 36A refer to the coordinates given to the position of the soccer field 36A among the coordinates given to the three-dimensional area image 32. Here, the coordinates indicating a position higher than the actual position are, for example, coordinates indicating a position higher than the actual position by a distance equivalent to the average height of a general adult. Note that the soccer field 36A is an example of a "specific area" according to the technology of the present disclosure.

[0048] As an example, and as shown in FIG. 3, the user device 12 includes a display 18 , a computer 40 , an imaging device 42 , a transceiver device 44 , a speaker 46 , a microphone 48 , an accepting device 50 , and a bus 52 .

[0049] The computer 40 includes a CPU 40A, an NVM 40B, and a RAM 40C, which are connected to each other via a bus 52. In the example shown in Fig. 3, for convenience of illustration, one bus is shown as the bus 52, but multiple buses may be used. The bus 52 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, and the like.

[0050] The CPU 40A controls the entire user device 12. The NVM 40B stores various parameters and various programs. An example of the NVM 40B is an EEPROM. The RAM 40C temporarily stores various information. The RAM 40C is used as a work memory by the CPU 40A.

[0051] The imaging device 42 is an imaging device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The imaging device 42 is connected to a bus 52, and the CPU 40A controls the imaging device 42. A captured image obtained by imaging the imaging device 42 is acquired by the CPU 40A via the bus 52.

[0052] The transmission / reception device 44 is connected to the bus 52. The transmission / reception device 44 is a device including a communication processor (not shown) and an antenna, and transmits and receives various information to and from the information processing device 10 via a base station (not shown) under the control of the CPU 40A. That is, the CPU 40A transmits and receives various information to and from the information processing device 10 via the transmission / reception device 44.

[0053] The speaker 46 converts an electrical signal into sound. The speaker 46 is connected to the bus 52. The speaker 46 receives an electrical signal output from the CPU 40A via the bus 52, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the user device 12.

[0054] The microphone 48 converts the collected sound into an electrical signal. The microphone 48 is connected to a bus 52. The electrical signal obtained by converting the sound collected by the microphone 48 is acquired by the CPU 40A via the bus 52.

[0055] The receiving device 50 receives instructions from the user 13, etc. Examples of the receiving device 50 include the touch panel 20 and hard keys (not shown). The receiving device 50 is connected to the bus 52, and the instructions received by the receiving device 50 are acquired by the CPU 40A.

[0056] The NVM 40B stores a user device side processing program 54. The CPU 40A reads the user device side processing program 54 from the NVM 40B, and executes the user device side processing program 54 on the RAM 40C to perform user device side processing (see FIG. 11).

[0057] 4 as an example, an observation mode in which the user 13 observes the inside of the soccer stadium 36 (hereinafter also simply referred to as the "observation mode") is defined by a viewpoint position 56, a line of sight direction 58, and a viewing angle θ of the user 13. The viewpoint position 56 corresponds to a position in real space from which the user 13 observes the inside of the soccer stadium 36, and is an example of a "reference position" and an "observation position" according to the technology of the present disclosure.

[0058] The viewpoint position 56 is a position corresponding to the position of the imaging device 42 mounted on the user device 12, the line of sight direction 58 is a direction corresponding to the direction of the optical axis of the imaging optical system (not shown) included in the imaging device 42, and the viewing angle θ is an angle corresponding to the angle of view of the imaging device 42. In the information processing system 2, an area observed by the user 13 in real space (actual space) is identified from a captured image obtained by capturing an image of the inside of the soccer stadium 36 by the imaging device 42. Then, in the information processing system 2, a viewpoint position different from the viewpoint position 56 is set using the captured image, and the user 13 views a subject image showing a subject existing in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from the set viewpoint position. In order to realize such viewing, the information processing system according to the first embodiment performs the user device side processing and image generation processing described below.

[0059] As an example, as shown in FIG. 5, in the user device side processing, the CPU 40A acquires a live view image obtained by capturing an image by the imaging device 42. The live view image is an image showing a specified area in the soccer stadium 36. Here, the specified area in the soccer stadium 36 refers to an area defined by, for example, a viewpoint position 56, a line of sight direction 58, and a viewing angle θ. The live view image is an example of "an image showing the state of a three-dimensional area when the three-dimensional area is observed in an observation state" according to the technology of the present disclosure. The CPU 40A generates a reference image 60 using the acquired live view image.

[0060] The reference image 60 is an example of a "designated area image" according to the technology of the present disclosure. The reference image 60 is an image showing the state of the inside of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from the viewpoint position 56 (see FIG. 4). The reference image 60 is an image based on a live view image. In the example shown in FIG. 5, an image in which a cross-shaped target mark 60A is superimposed on the live view image is shown as an example of the reference image 60. The target mark 60A is a mark that displaces in the reference image 60 according to an instruction given by the user 13, and indicates a position indicated by the user 13 as a viewpoint position from which the user 13 observes the inside of the soccer stadium 36. In other words, the target mark 60A is a mark that can specify the position indicated by the user 13 in the reference image 60. Note that the position of the target mark 60A in the reference image 60, i.e., the position of the target mark 60A superimposed on the live view image, is an example of the "instruction position", the "specific position in the captured image", and the "first mark" according to the technology of the present disclosure.

[0061] The home position of the target mark 60A is the center of the reference image 60. In the example shown in Fig. 5, the center of the target mark 60A is located at the center of the reference image 60. The CPU 40A causes the display 18 to display the reference image 60.

[0062] As an example, as shown in FIG. 6, in the user device side processing, the CPU 40A When the touch panel 20 receives a change instruction to change the position of the target mark 60A while the image 60 is displayed on the display 18, the position of the target mark 60A in the reference image 60 is changed according to the change instruction. The change instruction is a swipe performed on the touch panel 20 on the target mark 60A displayed on the display 18. That is, the user 13 touches the target mark 60A via the touch panel 20 and slides the touched position on the touch panel 20 to indicate the destination of the change of the position of the target mark 60A. The CPU 40A updates the reference image 60 displayed on the display 18 to the reference image 60 in which the position of the target mark 60A has been changed. Note that when the reference image 60 is a live view image, instead of changing the position of the target mark 60A on the display 18 by touching it, the user 13 may move the user device 12 to move the position of the target mark 60A relative to the reference image 60.

[0063] As an example, as shown in FIG. 7, in the user device side processing, when a confirmation instruction, which is an instruction to confirm the position of a target mark 60A in the reference image 60, is received by the touch panel 20 while the reference image 60 is displayed on the display 18, the CPU 40A generates a reference image 62 with a designated position. The reference image 62 with a designated position is an image in which designated position identification information 62A is added to the reference image 60. The designated position identification information 62A refers to information that can identify a position designated by the user 13 as a viewpoint position at which the user 13 observes the inside of the soccer stadium 36, that is, information that can identify the position of the target mark 60A in the reference image 60 (for example, information that can identify the position of a pixel corresponding to the center of the target mark 60A in the reference image 60). The CPU 40A transmits the reference image 62 with a designated position to the information processing device 10 via the transmission / reception device 44 (see FIG. 3).

[0064] As an example, as shown in FIG. 8, in the user device side processing, the CPU 40A acquires a virtual viewpoint image 64 generated by the information processing device 10. The virtual viewpoint image 64 is a moving image. However, this is merely an example, and the virtual viewpoint image 64 may be a still image. The CPU 40A displays the acquired virtual viewpoint image 64 on the display 18. Here, the CPU 40A does not simply display the virtual viewpoint image 64 on the display 18, but displays it on the display 18 as a new reference image 60.

[0065] The new reference image 60 is an image based on the virtual viewpoint image 64. That is, the new reference image 60 here refers to an image in which a target mark 60A is superimposed on the virtual viewpoint image 64. In the example shown in FIG. 8, an image in which a target mark 60A is superimposed on the virtual viewpoint image 64 so that the target mark 60A is located at the center of the virtual viewpoint image 64 is shown as the new reference image 60 displayed on the display 18. In addition, although described in detail later, the virtual viewpoint image 64 includes a mark 61 (see FIG. 10) that can specify a position within the virtual viewpoint image 64 that corresponds to the designated position used to generate the virtual viewpoint image 64, and the mark 61 is also included in the new reference image 60.

[0066] The new reference image 60 is an example of a "designated area image" according to the technology of the present disclosure. The virtual viewpoint image 64 is an example of an "image showing the state of the three-dimensional area when the three-dimensional area is observed in an observation state" according to the technology of the present disclosure. The position of the target mark 60A superimposed on the virtual viewpoint image 64 is an example of a "designated position" and a "specific position in the virtual viewpoint image" according to the technology of the present disclosure.

[0067] 9, in the image generation process, the CPU 22A acquires a reference image 62 with a designated position from the user device 12. The CPU 22A also acquires a three-dimensional area image 32 from the NVM 22B.

[0068] In the image generation process, the CPU 22A acquires a subject image showing a subject present within the soccer stadium 36 when observing the soccer stadium 36 from a viewpoint position determined based on coordinates within the soccer stadium 36 corresponding to the position of the target mark 60A in the reference image 60.

[0069] 10 as an example, first, the CPU 22A compares the reference image 62 with the pointing position acquired from the user device 12 with the three-dimensional area image 32 acquired from the NVM 22B to identify matching feature points between the reference image 62 with the pointing position and the three-dimensional area image 32. This identifies which pixel in the reference image 62 with the pointing position corresponds to which pixel in the three-dimensional area image 32.

[0070] The CPU 22A derives coordinates within the soccer stadium 36 corresponding to the indicated position from the three-dimensional area image 32 based on the observation manner in which the user 13 observes the soccer stadium 36 (in the example shown in FIG. 4, the viewpoint position 56, the line of sight direction 58, and the field of view angle θ) and the position of the target mark 60A within the reference image 60 (hereinafter, in this first embodiment, this position will also be simply referred to as the “indicated position”).

[0071] The observation manner in which the user 13 observes the inside of the soccer stadium 36 is determined according to the viewpoint position 56, and changes according to the displacement of the viewpoint position 56. Since the observation manner in which the user 13 observes the inside of the soccer stadium 36 is represented by the reference image 62 with the designated position, the CPU 22A derives coordinates in the soccer stadium 36 corresponding to the designated position from the three-dimensional area image 32 based on the correspondence between the reference image 62 with the designated position and the three-dimensional area image 32.

[0072] Specifically, the CPU 22A uses the comparison result between the reference image 62 with the indicated position and the three-dimensional area image 32 (for example, the comparison result as to which pixel in the reference image 62 with the indicated position corresponds to which pixel in the three-dimensional area image 32) to extract the coordinates of the position corresponding to the indicated position from the three-dimensional area image 32.

[0073] The CPU 22A generates a virtual viewpoint image 64 using a viewpoint position determined based on coordinates extracted from the three-dimensional area image 32. The viewpoint position determined based on coordinates extracted from the three-dimensional area image 32 refers to a position within the soccer stadium 36 that is identified by the coordinates extracted from the three-dimensional area image 32, for example.

[0074] The virtual viewpoint image 64 is a type of subject image showing a subject present in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position determined based on coordinates extracted from the three-dimensional area image 32. In this case, the observation mode (observation mode in which the user 13 virtually observes the inside of the soccer stadium 36), i.e., the viewpoint position, line of sight direction, and viewing angle used to generate the virtual viewpoint image 64, are specified by, for example, the viewpoint position determined based on coordinates extracted from the three-dimensional area image 32, the line of sight direction specified in advance by the user 13, etc., and the viewing angle specified in advance by the user 13, etc.

[0075] In addition, when the virtual viewpoint image 64 includes a pointing position identified from the pointing position identification information 62A of the reference image 62 with pointing position, the CPU 22A assigns a mark 61 (in the example shown in Figure 8, a dashed cross mark) that can identify the pointing position within the virtual viewpoint image 64 to the pointing position.

[0076] The mark 61 is an example of a "second mark" according to the technology of the present disclosure. The viewpoint position, line of sight direction, and viewing angle used to generate the virtual viewpoint image 64 are an example of an "observation mode" according to the technology of the present disclosure. The area defined by the viewpoint position, line of sight direction, and viewing angle used to generate the virtual viewpoint image 64 is an example of an "area specified within a three-dimensional area" according to the technology of the present disclosure. The viewpoint position used to generate the virtual viewpoint image 64 corresponds to a position from which the user 13 virtually observes the inside of the soccer stadium 36, and is an example of an "observation position" according to the technology of the present disclosure.

[0077] Here, a moving image using 3D polygons generated based on a plurality of captured images obtained by capturing images of the inside of the soccer stadium 36 by the plurality of imaging devices 30 is applied as an example of the virtual viewpoint image 64. Note that the moving image is merely an example, and a still image may also be used.

[0078] The CPU 22A transmits the virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24 (see FIG. 1). The virtual viewpoint image 64 thus transmitted is received by the user device 12 and displayed on the display 18 as a new reference image 60 (see FIG. 8).

[0079] Next, the operation of the information processing system 2 will be described.

[0080] First, an example of the flow of user device side processing performed by the CPU 40A of the user device 12 will be described with reference to FIG.

[0081] In the user device-side process shown in FIG. 11, first, in step ST10, the CPU 40A acquires a live view image from the imaging device 42, and then the user device-side process proceeds to step ST12.

[0082] In step ST12, the CPU 40A generates a reference image 60 based on the live view image acquired in step ST10, and then the user device side processing proceeds to step ST14.

[0083] In step ST14, the CPU 40A causes the reference image 60 generated in step ST12 to be displayed on the display 18, and thereafter, the user device side processing proceeds to step ST16.

[0084] In step ST16, the CPU 40A determines whether the designated position has been confirmed. Here, if a confirmation instruction is received by the touch panel 20, it is determined that the designated position has been confirmed, and if a confirmation instruction is not received by the touch panel 20, it is determined that the designated position has not been confirmed. In step ST16, if the designated position has not been confirmed, the determination is negative, and the user device-side processing proceeds to step ST28. In step ST16, if the designated position has been confirmed, the determination is positive, and the user device-side processing proceeds to step ST18.

[0085] In step ST28, the CPU 40A determines whether a condition for terminating the user device side process (hereinafter referred to as a "user device side process termination condition") is satisfied. A first example of the user device side process termination condition is that an instruction to terminate the user device side process is accepted by the accepting device 50. A second example of the user device side process termination condition is that a first predetermined time (e.g., 60 minutes) has elapsed since the execution of the user device side process was started. A third example of the user device side process termination condition is that the processing capacity of the CPU 40A has decreased to below a reference level.

[0086] In step ST28, if the user device side process end condition is not satisfied, the determination is negative, and the user device side process proceeds to step ST10. In step ST28, if the user device side process end condition is satisfied, the determination is positive, and the user device side process ends.

[0087] In step ST18, the CPU 40A generates a reference image 62 with a pointing position based on the reference image 60 generated in step ST12 or a reference image 60 in which a target mark 60A is added to a virtual viewpoint image 64 received by the transmission / reception device 44 in step ST20 described below. Then, the CPU 40A transmits the generated reference image 62 with a pointing position to the information processing device 10 via the transmission / reception device 44. After the process of step ST18 is executed, the user device side process proceeds to step ST20.

[0088] In step ST20, the CPU 40A judges whether or not the virtual viewpoint image 64 transmitted from the information processing device 10 has been received by the transmission / reception device 44 by executing the process of step ST60 of the image generation process shown in Fig. 12. If the virtual viewpoint image 64 has not been received by the transmission / reception device 44 in step ST20, the judgment is negative, and the judgment of step ST20 is performed again. If the virtual viewpoint image 64 has been received by the transmission / reception device 44 in step ST20, the judgment is positive, and the user device side processing proceeds to step ST22.

[0089] In step ST22, the CPU 40A causes the display 18 to display the virtual viewpoint image 64 received by the transmission / reception device 44 in step ST20 as a new reference image 60, and then the user device side processing proceeds to step ST24.

[0090] In step ST24, the CPU 40A judges whether the designated position has been confirmed. If the designated position has not been confirmed in step ST24, the judgment is negative, and the user device side processing proceeds to step ST26. If the designated position has been confirmed in step ST24, the judgment is positive, and the user device side processing proceeds to step ST18.

[0091] In step ST26, the CPU 40A judges whether the user device side processing end condition is satisfied. If the user device side processing end condition is not satisfied in step ST26, the judgment is negative, and the user device side processing proceeds to step ST24. If the user device side processing end condition is satisfied in step ST26, the judgment is positive, and the user device side processing ends.

[0092] Next, an example of the flow of an image generation process performed by the CPU 22A of the information processing device 10 will be described with reference to Fig. 12. Note that the flow of the image generation process shown in Fig. 12 is an example of an "information processing method" according to the technique of the present disclosure.

[0093] In the image generation process shown in Fig. 12, first, in step ST50, the CPU 22A determines whether or not the reference image 62 with the pointing position transmitted by executing the process of step ST18 of the user device side process shown in Fig. 11 has been received by the transmission / reception device 24. If the reference image 62 with the pointing position has not been received by the transmission / reception device 24 in step ST50, the determination is negative, and the image generation process proceeds to step ST62. If the reference image 62 with the pointing position has been received by the transmission / reception device 24 in step ST50, the determination is positive, and the image generation process proceeds to step ST52.

[0094] In step ST52, the CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and then the image generation process proceeds to step ST54.

[0095] In step ST54, the CPU 22A compares the reference image 62 with the designated position received by the transmitting / receiving device 24 in step ST50 with the three-dimensional area image 32 acquired in step ST52, and then the image generating process proceeds to step ST56.

[0096] In step ST56, the CPU 22A uses the comparison result between the reference image 62 with the designated position and the three-dimensional area image in step ST54 to extract coordinates from the three-dimensional area image 32 corresponding to the designated position identified from the designated position identification information 62A of the reference image 62 with the designated position, and then the image generation process proceeds to step ST58.

[0097] In step ST58, the CPU 22A generates a virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted in step ST56, and thereafter, the image generation process proceeds to step ST60.

[0098] In step ST60, the CPU 22A transmits the virtual viewpoint image 64 generated in step ST58 to the user device 12 via the transmission / reception device 24, and then the image generation process proceeds to step ST62.

[0099] In step ST62, the CPU 22A determines whether or not a condition for terminating the image generation process (hereinafter referred to as the "image generation process termination condition") has been satisfied. A first example of the image generation process termination condition is that an instruction to terminate the image generation process is given to the information processing device 10 from an administrator of the information processing device 10 or the like. A second example of the image generation process termination condition is that a second predetermined time (e.g., 10 hours) has elapsed since the execution of the image generation process was started. A third example of the image generation process termination condition is that the processing capacity of the CPU 22A has decreased to below a reference level.

[0100] In step ST62, if the image generation process end condition is not satisfied, the determination is negative and the image generation process proceeds to step ST50. If the image generation process end condition is satisfied, the determination is positive and the image generation process ends.

[0101] As described above, in the information processing system 2, the CPU 22A acquires a virtual viewpoint image 64 showing a subject present in the soccer stadium 36 when the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates in the soccer stadium 36 corresponding to a designated position designated in a reference image 60 (see FIG. 6) showing the state inside the soccer stadium 36 when the user 13 observes the soccer stadium 36 (see FIG. 4) from the viewpoint position 56 (see FIG. 4). The virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can observe the state inside the soccer stadium 36 from various positions.

[0102] Furthermore, in the information processing system 2, the CPU 22A derives coordinates within the soccer stadium 36 corresponding to the indicated position based on the observation manner in which the user 13 observes the inside of the soccer stadium 36 and the indicated position. Therefore, according to this configuration, it is possible to identify which position within the soccer stadium 36 the position indicated by the user 13 is.

[0103] Furthermore, in the information processing system 2, the observation manner in which the user 13 observes the inside of the soccer stadium 36 is determined according to the position in the real space or the virtual space from which the user 13 observes the inside of the soccer stadium 36. Therefore, when the position from which the user 13 observes the inside of the soccer stadium 36 changes, the observation manner in which the user 13 observes the inside of the soccer stadium 36 also changes accordingly. In this case, too, the CPU 22A derives the coordinates in the soccer stadium 36 corresponding to the indicated position based on the observation manner in which the user 13 observes the inside of the soccer stadium 36 and the indicated position. Therefore, according to this configuration, even if the observation manner changes in accordance with the displacement of the position from which the user 13 observes the inside of the soccer stadium 36, it is possible to identify which position in the soccer stadium 36 the position indicated by the user 13 is.

[0104] Furthermore, in the information processing system 2, the CPU 22A derives coordinates in the soccer stadium 36 corresponding to the indicated position based on the correspondence between the live view image or virtual viewpoint image 64 and the three-dimensional area image 32. Therefore, according to this configuration, it is possible to specify with high accuracy which position in the soccer stadium 36 the position indicated by the user 13 is, compared to a case in which the position in the soccer stadium 36 the position indicated by the user 13 is estimated based only on information obtained by visual inspection from the live view image or virtual viewpoint image 64 and human intuition.

[0105] Furthermore, in the information processing system 2, an image based on a live view image and an image based on a virtual viewpoint image 64 are used as the reference image 60. Therefore, according to this configuration, the user 13 can specify the viewpoint position while checking the state in the real space, or specify the viewpoint position while checking the state in the virtual space.

[0106] Furthermore, in the information processing system 2, the position of the target mark 60A in the live view image or the virtual viewpoint image 64 is the position specified by the user 13. The position of the target mark 60A is changed in accordance with a change instruction (see FIG. 6) given by the user 13, and is confirmed in accordance with a confirmation instruction (see FIG. 7) given by the user 13. Therefore, according to this configuration, the position intended by the user 13 can be set as the viewpoint position.

[0107] Furthermore, in the information processing system 2, a target mark 60A is used as a mark capable of identifying an indicated position in the reference image 60, and the target mark 60A is displayed on the display 18 of the user device 12 in a state in which it is included in the reference image 60. Therefore, according to this configuration, the user 13 can visually recognize the indicated position in the reference image 60.

[0108] Moreover, in the information processing system 2, the virtual viewpoint image 64 includes a mark 61. The mark 61 is a mark that allows the pointing position used to generate the virtual viewpoint image 64 to be identified within the virtual viewpoint image 64. Therefore, according to this configuration, the user 13 can infer the pointing position used to generate the virtual viewpoint image 64 from the virtual viewpoint image 64.

[0109] Furthermore, in the information processing system 2, the coordinates relating to the position of the soccer field 36A shown in the three-dimensional area image 32 are set to be coordinates indicating a position higher than the actual position of the soccer field 36A. Therefore, according to this configuration, it is possible to prevent the viewpoint position from being set on the ground of the soccer field 36A.

[0110] Furthermore, in the information processing system 2, the CPU 22A detects the designated position based on the reference image 60. That is, the position of the target mark 60A is detected as the designated position by the CPU 22A. Therefore, according to this configuration, the designated position can be specified with high accuracy compared to the case where the designated position is estimated only based on information obtained visually from the live view image or the virtual viewpoint image 64 and human intuition.

[0111] In the above-described first embodiment, an example in which the CPU 22A generates the virtual viewpoint image 64 in the image generation process and transmits it to the user device 12 has been described, but the technology of the present disclosure is not limited to this. For example, when the imaging device 30 is installed at a position that coincides with the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32, a captured image obtained by capturing an image using the imaging device 30 installed at the position that coincides with the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32 may be transmitted from the information processing device 10 to the user device 12 as a substitute image for the virtual viewpoint image 64.

[0112] In the above first embodiment, an example has been described in which the CPU 22A generates a new virtual viewpoint image 64 and transmits it to the user device 12 every time it acquires the reference image 62 with the designated position, but the technology of the present disclosure is not limited to this. For example, if an object image showing an object present in the soccer stadium 36 when the soccer stadium 36 is observed from a position whose distance from the designated position identified from the designated position identification information 62A of the reference image 62 with the designated position is within a threshold range is stored in the storage area, the CPU 22A may acquire the object image from the storage area and transmit the acquired object image to the user device 12.

[0113] In this case, as shown in FIG. 13 as an example, the CPU 22A extracts coordinates corresponding to the designated position identified from the designated position identification information 62A of the reference image 62 with the designated position from the three-dimensional area image 32, as in the first embodiment. Then, the CPU 22A determines whether or not a virtual viewpoint image 64 associated with coordinates within a neighborhood range of the coordinates extracted from the three-dimensional area image 32 is stored in the NVM 22B. Here, the neighborhood range refers to a range within a radius of, for example, 2 meters. The radius of 2 meters is an example of a "threshold" according to the technology of the present disclosure. The NVM 22B is an example of a "storage area" according to the technology of the present disclosure. The virtual viewpoint image 64 is an example of an "object image" according to the technology of the present disclosure, and the object shown by the virtual viewpoint image 64 is an example of an "object" according to the technology of the present disclosure.

[0114] If the virtual viewpoint image 64 associated with the coordinates within the vicinity range of the coordinates extracted from the three-dimensional area image 32 by the CPU 22A is not stored in the NVM 22B, the CPU 22A generates the virtual viewpoint image 64 in a manner similar to that of the first embodiment, and transmits the generated virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24. The CPU 22A also associates the coordinates extracted from the three-dimensional area image 32 with the generated virtual viewpoint image 64, and stores the virtual viewpoint image 64 associated with the coordinates in the NVM 22B.

[0115] On the other hand, when the virtual viewpoint image 64 associated with the coordinates within the vicinity range of the coordinates extracted from the three-dimensional area image 32 by the CPU 22A is stored in the NVM 22B, as shown in Fig. 14 as an example, the CPU 22A acquires from the NVM 22B the virtual viewpoint image 64 associated with the coordinates closest to the coordinates extracted from the three-dimensional area image 32. Then, the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24. This makes it possible to provide the virtual viewpoint image 64 to the user 13 more quickly than when the CPU 22A generates a new virtual viewpoint image 64 every time it acquires the reference image 62 with the designated position.

[0116] In the examples shown in Figures 13 and 14, an example is described in which a virtual viewpoint image 64 is generated and transmitted to the user device 12 or stored in NVM 22B, but the technology of the present disclosure is not limited to this, and a captured image may be transmitted to the user device 12 or stored in NVM 22B together with or instead of the virtual viewpoint image 64.

[0117] [Second embodiment] In the above first embodiment, a case was described in which the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates in the soccer stadium 36 corresponding to a position indicated in the reference image 60, but in the present second embodiment, a case is described in which the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates in the soccer stadium 36 corresponding to a position indicated in the soccer stadium 36 to be observed. In the present second embodiment, the same components as in the above first embodiment are denoted by the same reference numerals, their explanation is omitted, and only the parts different from the above first embodiment are explained.

[0118] As an example, as shown in FIG. 15, an information processing system 66 includes an information processing device 10, a user device 12, and an HMD 68.

[0119] The HMD 68 includes an HMD body 70 and a band 72. The band 72 is a stretchable member formed in a band shape from one end to the other end of the HMD body 70. The HMD 68 has an outer shape formed into a circular ring by the HMD body 70 and the band 72, and is fixed in a state of intimate contact with the upper half of the head of the user 13.

[0120] The HMD main body 70 has a display 74, an HMD camera 76, a computer 78, a reception device 80, and a transmission / reception device 82. The display 74 has a screen (not shown) and a projection unit (not shown). The screen is made of a transparent material, and the user 13 visually recognizes the real space with the naked eye through the screen. In other words, the HMD 68 is a see-through type HMD. The HMD main body 70 does not necessarily have to include the computer 78, and the computer 78 may be provided separately from the HMD main body 70. In that case, the HMD main body 70 may only have a function of displaying data received from the computer 78 via the transmission / reception device 82 and transmitting data related to an image obtained by capturing an image with the HMD camera 76 to the computer 78. The HMD camera 76 may also be provided separately from the HMD main body 70. For example, the HMD camera 76 may be a camera that is detachable from the HMD main body 70.

[0121] The screen is located directly opposite the eyes of the user 13, and an image is projected onto the inner surface of the screen (the surface facing the user 13) by the projection unit. The projection unit is a well-known device and will not be described in detail, but it is a device having a display element such as a liquid crystal display for displaying an image, and a projection optical system for projecting the image displayed on the display element toward the inner surface of the screen. The screen is realized by a half mirror that reflects the image projected by the projection unit and transmits light in the real space. The projection unit projects an image onto the inner surface of the screen at a preset frame rate (for example, 60 frames / second). The image is reflected by the inner surface of the screen and enters the eyes of the user 13. This allows the user 13 to visually recognize the image.

[0122] The HMD camera 76 is an imaging device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The HMD camera 76 is located in front of the forehead of the user 13, and captures an image in front of the user 13.

[0123] The computer 78 includes a CPU 78A, an NVM 78B, and a RAM 78C, and the CPU 78A, the NVM 78B, and the RAM 78C are connected via a bus 84. In the example shown in Fig. 15, for convenience of illustration, one bus is shown as the bus 84, but multiple buses may be used. In addition, the bus 84 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, and the like.

[0124] The CPU 78A controls the entire HMD main body 70. The NVM 78B stores various parameters and various programs. An example of the NVM 78B is an EEPROM. Various information is temporarily stored in the RAM 78C. The RAM 78C is used as a work memory by the CPU 78A.

[0125] The display 74 is connected to the bus 84. Specifically, the above-mentioned projection unit is connected to the bus 84. The display 74 displays various information under the control of the CPU 78A.

[0126] The HMD camera 76 is connected to a bus 84, and the CPU 78A controls the HMD camera 76. A captured image obtained by capturing an image with the HMD camera 76 is acquired via the bus 84 by the CPU 78A.

[0127] The transmission / reception device 82 is connected to the bus 84. The transmission / reception device 82 is a device including a communication processor (not shown) and an antenna, and transmits and receives various information to and from the information processing device 10 via a base station (not shown) under the control of the CPU 78A. That is, the CPU 78A transmits and receives various information to and from the information processing device 10 via the transmission / reception device 82.

[0128] The receiving device 80 is a device including at least one hard key, and receives instructions from the user 13. The receiving device 80 is connected to a bus 84, and the CPU 78A obtains the instructions received by the receiving device 80.

[0129] The NVM 78B stores an HMD-side processing program 85. The CPU 78A reads out the HMD-side processing program 85 from the NVM 78B, and executes the HMD-side processing program 85 on the RAM 78C to perform HMD-side processing (see FIG. 28).

[0130] 16 as an example, in the HMD-side processing, the CPU 78A acquires an HMD image 86 (see FIG. 17) from the HMD camera 76. The HMD image 86 is, for example, a live view image. The CPU 78A causes the display 74 to display the HMD image 86.

[0131] 17 as an example, the user 13 can observe the real space through the entire display 74. An HMD image 86 is displayed on the display 74. The HMD image 86 is displayed by the user 13 through the display 74 in a state where it is superimposed on a part of the real space area. Since light from the real space also passes through the display area of ​​the HMD image 86, the user 13 can observe the real space through the display area of ​​the HMD image 86.

[0132] As an example, as shown in FIG. 18, when the finger of the user 13 enters the angle of view of the HMD camera 76, the finger of the user is captured by the HMD camera 76 and is displayed as an image in the HMD image 86. By pointing the finger within the angle of view of the HMD camera 76, the user 13 provisionally designates the position pointed to by the finger as the viewpoint position from which the user 13 observes the inside of the soccer stadium 36. Note that the direction in which the finger is actually pointed is shifted from the line of sight of the user 13, so the point where the finger is pointed is not the position intended by the user 13 as the viewpoint position. The position intended by the user 13 as the viewpoint position is the end of the line of sight that passes through the fingertip of the user 13. The direction of the optical axis OA of the imaging optical system of the HMD camera 76 is approximately the same as the line of sight of the user 13. Therefore, the information processing system 66 determines the position that meets the optical axis OA within the soccer stadium 36 when the center of the angle of view and the position of the fingertip of the user 13 coincide, that is, the point (gazing point) at which the user 13 is currently observing the soccer stadium 36, as the viewpoint position from which the user 13 observes the soccer stadium 36, as the position provisionally indicated by the user 13 (hereinafter also referred to as the ``provisional indicated position'').

[0133] As an example, as shown in Fig. 19, in the HMD-side processing, the CPU 78A detects the finger of the user 13 using an HMD image 86 obtained by capturing an image by the HMD camera 76. The CPU 78A generates an HMD image 88 (see Fig. 20) with a tentative pointing position when the fingertip of the user 13 stops at the center of the angle of view. When the fingertip of the user 13 stops at the center of the angle of view, it means, for example, that the state in which the fingertip of the user 13 stops at the center of the angle of view continues for a time (e.g., 3 seconds) specified in advance. The CPU 78A transmits the generated HMD image 88 with a tentative pointing position to the information processing device 10 via the transmission / reception device 82 (see Fig. 15).

[0134] 20, an HMD image 88 with a tentative pointing position is an image in which tentative pointing position identification information 88A is added to an HMD image 86. The tentative pointing position identification information 88A refers to information capable of identifying a tentative pointing position in the HMD image 86 (for example, information capable of identifying the position of a pixel corresponding to the tentative pointing position in the HMD image 86, i.e., information capable of identifying the position of a pixel corresponding to the position of a fingertip captured in the HMD image 86).

[0135] 21 as an example, in the image generation process of the information processing device 10, the CPU 22A acquires an HMD image 88 with a tentative pointing position from the HMD 68. The CPU 22A acquires, from the imaging device 30, a captured image showing a subject including a tentative pointing position identified from tentative pointing position identification information 88A of the HMD image 88 with a tentative pointing position, as a different viewpoint position image 90 (see FIG. 22) showing the state of the inside of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position different from the current viewpoint position of the user 13.

[0136] The CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and generates a different viewpoint position image 92 (see FIG. 22) with a designation position candidate by referring to the acquired three-dimensional area image 32. Specifically, the CPU 22A first identifies a feature point that matches between the HMD image 88 with the tentative designation position and the three-dimensional area image 32, thereby identifying the position of the optical axis OA in the soccer stadium 36. Next, the CPU 22A compares the HMD image 88 with the tentative designation position with the three-dimensional area image 32, and extracts coordinates of the tentative designation position in the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. Next, the CPU 22A generates a plurality of designation position candidates. The plurality of designation position candidates are positions defined on the optical axis OA at a predetermined interval (for example, 5-meter intervals on a real space scale) including the tentative designation position. Each of the plurality of designation position candidates is associated with a coordinate obtained from the three-dimensional area image 32. Then, the CPU 22A generates a different viewpoint position image 92 with designation position candidates (see FIG. 22) by adding information such as a plurality of designation position candidates to the different viewpoint position image 90. The CPU 22A transmits the different viewpoint position image 92 with designation position candidates to the HMD 68 via the transmission / reception device 24 (see FIG. 1).

[0137] The optical axis OA is an example of a "first line" according to the technology of the present disclosure, and the tentatively indicated position is an example of a "gazing point" according to the technology of the present disclosure.

[0138] 22, the other viewpoint position image 92 with the pointing position candidate is an image in which a plurality of dot marks 92A and a message 92B are superimposed on the other viewpoint position image 90. The plurality of dot marks 92A are arranged at predetermined intervals on the image showing the optical axis OA, and each dot mark 92A is a mark capable of identifying a pointing position candidate. Note that it is not essential to display the image showing the optical axis OA, and the image showing the optical axis OA does not have to be displayed.

[0139] Each dot mark 92A is associated with coordinates obtained from the three-dimensional area image 32 as coordinates by which the position of the designated position candidate can be specified. The message 92B is a message that prompts the user 13 to select a designated position candidate, and in the example shown in Fig. 22, a message "Please specify any dot (position)" is displayed as an example of the message 92B.

[0140] 23 as an example, in the HMD-side processing, the CPU 78A acquires another viewpoint position image 92 with the pointing position candidate from the information processing device 10. Then, as shown in FIG 24 as an example, the CPU 78A causes the display 74 to display the other viewpoint position image 92 with the pointing position candidate.

[0141] The user 13 positions his / her fingertip on any one of the multiple dot marks 92A included in the other viewpoint position image 92 with the designated position candidate, thereby indicating the observation position intended by the user 13 within the soccer stadium 36. That is, the observation position intended by the user 13 is determined by the user 13 indicating a plurality of designated position candidates provided on the optical axis OA. Note that the observation position intended by the user 13 here is an example of the "designated position", the "observation position", and the "designated position on the first line" according to the technology of the present disclosure.

[0142] 25 as an example, in the HMD-side processing, the CPU 78A acquires an HMD image 86 from the HMD camera 76 and detects the finger of the user 13 using the acquired HMD image 86. When the fingertip of the user 13 is located on any of the dot marks 92A, the CPU 78A transmits information including the coordinates associated with the dot mark 92A on which the fingertip of the user 13 is located as pointing position identification information 94 to the information processing device 10 via the transmission / reception device 82 (see FIG. 15). The pointing position identification information 94 is information capable of identifying a pointing position candidate selected by the user 13 via the dot mark 92A, that is, information capable of identifying a position designated by the user 13 as a viewpoint position from which the user 13 observes the inside of the soccer stadium 36.

[0143] 26 as an example, in the image generation process of the information processing device 10, the CPU 22A acquires pointing position identification information 94 from the HMD 68. The CPU 22A extracts coordinates from the pointing position identification information 94, and generates a virtual viewpoint image 64 (see FIG. 10) using a viewpoint position determined based on the extracted coordinates. The CPU 22A transmits the generated virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24. As a result, in the user device 12, the virtual viewpoint image 64 is displayed on the display 18, similarly to the first embodiment.

[0144] Note that, although an example is given here in which the virtual viewpoint image 64 is transmitted to the user device 12, the technology of the present disclosure is not limited to this, and the virtual viewpoint image 64 may be transmitted to the HMD 68 and displayed on the display 74 of the HMD 68.

[0145] Next, the operation of the information processing system 66 will be described.

[0146] First, an example of the flow of user device side processing performed by the CPU 40A of the user device 12 will be described with reference to FIG.

[0147] In the example shown in Fig. 27, first, in step ST100, the CPU 40A determines whether or not the virtual viewpoint image 64 transmitted by executing the process of step ST214 of the image generation process shown in Fig. 30 has been received by the transmission / reception device 44. If the virtual viewpoint image 64 has not been received by the transmission / reception device 44 in step ST100, the determination is negative, and the user device-side processing proceeds to step ST104. If the virtual viewpoint image 64 has been received by the transmission / reception device 44 in step ST100, the determination is positive, and the user device-side processing proceeds to step ST102.

[0148] In step ST102, the CPU 40A causes the display 18 to display the virtual viewpoint image 64 received by the transmission / reception device 44 in step ST100, and then the user device side processing proceeds to step ST104.

[0149] In step ST104, the CPU 40A judges whether the user device side processing end condition is satisfied. If the user device side processing end condition is not satisfied in step ST104, the judgment is negative, and the user device side processing proceeds to step ST100. If the user device side processing end condition is satisfied in step ST104, the judgment is positive, and the user device side processing ends.

[0150] Next, an example of the flow of HMD-side processing performed by the CPU 78A of the HMD 68 will be described with reference to FIGS.

[0151] In the HMD-side processing shown in FIG. 28, first, in step ST150, the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD-side processing proceeds to step ST152.

[0152] In step ST152, the CPU 78A causes the display 74 to display the HMD image 86 acquired in step ST150, and thereafter, the HMD-side processing proceeds to step ST154.

[0153] In step ST154, the CPU 78A executes finger detection processing using the HMD image 86 acquired in step ST152. The finger detection processing refers to processing for detecting the finger of the user 13 using the HMD image 86. After the processing of step ST154 is executed, the HMD-side processing proceeds to step ST156.

[0154] In step ST156, the CPU 78A determines whether or not a finger of the user 13 is detected by the finger detection process in step ST154. If a finger of the user 13 is not detected by the finger detection process in step ST154 in step ST156, the determination is negative, and the process proceeds to step ST178 shown in Fig. 29. If a finger of the user 13 is detected by the finger detection process in step ST154 in step ST156, the determination is positive, and the HMD-side process proceeds to step ST158.

[0155] In step ST158, the CPU 78A determines whether or not the fingertip of the user 13 is stationary at the center of the angle of view of the HMD camera 76. In step ST158, if the fingertip of the user 13 is not stationary at the center of the angle of view of the HMD camera 76, the determination is negative, and the HMD-side processing proceeds to step ST150. In step ST158, if the fingertip of the user 13 is stationary at the center of the angle of view of the HMD camera 76, the determination is positive, and the HMD-side processing proceeds to step ST160.

[0156] In step ST160, the CPU 78A generates an HMD image 88 with a tentative designated position based on the HMD image 86 acquired in step ST150, and then the HMD-side processing proceeds to step ST162.

[0157] In step ST162, the CPU 78A transmits the HMD image 88 with the tentative designated position generated in step ST160 to the information processing device 10 via the transmission / reception device 82, and thereafter, the HMD-side processing proceeds to step ST164.

[0158] In step ST164, the CPU 78A determines whether or not the other viewpoint position image 92 with the designation position candidate transmitted by executing the process of step ST206 shown in Fig. 30 has been received by the transmission / reception device 82. If the other viewpoint position image 92 with the designation position candidate has not been received by the transmission / reception device 82 in step ST164, the determination is negative, and the determination of step ST164 is made again. If the other viewpoint position image 92 with the designation position candidate has been received by the transmission / reception device 82 in step ST164, the determination is positive, and the HMD-side processing proceeds to step ST166.

[0159] In step ST166, the CPU 78A causes the display 74 to display the other viewpoint position image 92 with the indication position candidate received by the transmission / reception device 82 in step ST164, and thereafter, the HMD-side processing proceeds to step ST168 shown in FIG.

[0160] In step ST168 shown in FIG. 29, the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD-side processing proceeds to step ST170.

[0161] In step ST170, the CPU 78A executes finger detection processing using the HMD image 86 acquired in step ST168, and thereafter, the HMD-side processing proceeds to step ST172.

[0162] In step ST172, the CPU 78A determines whether or not a finger of the user 13 is detected by the finger detection process of step ST170. If a finger of the user 13 is not detected by the finger detection process of step ST170 in step ST172, the determination is negative, and the process proceeds to step ST180. If a finger of the user 13 is detected by the finger detection process of step ST170 in step ST172, the determination is positive, and the HMD-side process proceeds to step ST174.

[0163] In step ST180, the CPU 78A determines whether or not a condition for terminating the HMD-side processing (hereinafter referred to as the "HMD-side processing termination condition") is satisfied. A first example of the HMD-side processing termination condition is that an instruction to terminate the HMD-side processing is accepted by the acceptance device 80. A second example of the HMD-side processing termination condition is that a third predetermined time (e.g., 60 minutes) has elapsed since the execution of the HMD-side processing was started. A third example of the HMD-side processing termination condition is that the processing capacity of the CPU 78A has decreased to below a reference level.

[0164] In step ST180, if the HMD-side processing end condition is not satisfied, the determination is negative, and the HMD-side processing proceeds to step ST168. In step ST180, if the HMD-side processing end condition is satisfied, the determination is positive, and the HMD-side processing ends.

[0165] In step ST174, the CPU 78A judges whether or not the fingertip (the fingertip of the user 13) detected in step ST172 is located at the dot mark 92A on the other viewpoint position image 92 with the indication position candidate displayed on the display 74. In step ST174, if the fingertip of the user 13 is not located at the dot mark 92A on the other viewpoint position image 92 with the indication position candidate, the judgment is negative, and the HMD-side processing proceeds to step ST168. In step ST174, if the fingertip of the user 13 is located at the dot mark 92A on the other viewpoint position image 92 with the indication position candidate, the judgment is positive, and the HMD-side processing proceeds to step ST176.

[0166] In step ST176, the CPU 78A transmits information including the coordinates associated with the dot mark 92A where the fingertip of the user 13 is located as instruction position identification information 94 to the information processing device 10 via the transmission / reception device 82, and then the HMD-side processing proceeds to step ST178.

[0167] In step ST178, the CPU 78A judges whether or not the HMD-side processing end condition is satisfied. If the HMD-side processing end condition is not satisfied in step ST178, the judgment is negative, and the HMD-side processing proceeds to step ST150 shown in Fig. 28. If the HMD-side processing end condition is satisfied in step ST178, the judgment is positive, and the HMD-side processing ends.

[0168] Next, an example of the flow of image generation processing performed by the CPU 22A of the information processing device 10 will be described with reference to FIG.

[0169] In the image generation process shown in Fig. 30, first, in step ST200, the CPU 22A determines whether or not the HMD image 88 with the temporary designation position transmitted by executing the process of step ST162 of the HMD-side process shown in Fig. 28 has been received by the transmission / reception device 24. If the HMD image 88 with the temporary designation position has not been received by the transmission / reception device 24 in step ST200, the determination is negative, and the image generation process proceeds to step ST208. If the HMD image 88 with the temporary designation position has been received by the transmission / reception device 24 in step ST200, the determination is positive, and the image generation process proceeds to step ST202.

[0170] In step ST202, the CPU 22A acquires, from the imaging device 30, a captured image showing a subject including the tentative pointing position identified from the tentative pointing position identification information 88A of the HMD image 88 with tentative pointing position received by the transmission / reception device 24 in step ST200, as a different viewpoint position image 90 showing the state of the inside of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position different from the current viewpoint position of the user 13. After the processing of step ST202 is executed, the image generation processing proceeds to step ST204.

[0171] In step ST204, the CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and generates a different viewpoint position image 92 with a designated position candidate by referring to the acquired three-dimensional area image 32. After that, the image generation process proceeds to step ST206.

[0172] In step ST206, the CPU 22A transmits the other viewpoint position image 92 with the indication position candidate generated in step ST202 to the HMD 68 via the transmission / reception device 24, and thereafter, the image generation process proceeds to step ST208.

[0173] In step ST208, the CPU 22A judges whether or not the designated position specifying information 94 transmitted by executing the process of step ST176 shown in Fig. 29 has been received by the transmission / reception device 24. If the designated position specifying information 94 has not been received by the transmission / reception device 24 in step ST208, the judgment is negative, and the image generation process proceeds to step ST216. If the designated position specifying information 94 has been received by the transmission / reception device 24 in step ST208, the judgment is positive, and the image generation process proceeds to step ST210.

[0174] In step ST210, the CPU 22A extracts coordinates from the designated position specifying information 94 received by the transmitting / receiving device 24 in step ST208, and then the image generating process proceeds to step ST212.

[0175] In step ST212, the CPU 22A generates the virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted from the designated position specifying information 94 in step ST210, and thereafter, the image generation process proceeds to step ST214.

[0176] In step ST214, the CPU 22A transmits the virtual viewpoint image 64 generated in step ST212 to the user device 12 via the transmission / reception device 24, and then the image generation process proceeds to step ST216.

[0177] In step ST216, the CPU 22A judges whether or not the image generation process end condition is satisfied. In step ST216, if the image generation process end condition is not satisfied, the judgment is negative and the image generation process proceeds to step ST200. If the image generation process end condition is satisfied, the judgment is positive and the image generation process ends.

[0178] As described above, in the information processing system 66, the CPU 22A acquires a virtual viewpoint image 64 that shows a subject present in the soccer stadium 36 when the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates in the soccer stadium 36 that correspond to a position specified within the soccer stadium 36. The virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, according to this configuration, it is possible for the user 13 to observe the state of the inside of the soccer stadium 36 from various positions.

[0179] Furthermore, in the information processing system 65, the CPU 22A acquires a virtual viewpoint image 64 showing a subject present in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a designated position (e.g., any one of a plurality of designated position candidates) on a line (e.g., on the optical axis OA) from a viewpoint (e.g., the position of the HMD camera 76 in the soccer stadium 36) at which the user 13 is observing to a gaze point (e.g., a tentative designated position) at which the user 13 is gazing, and the virtual viewpoint image 64 is displayed on the display 18 of the user device 12. Therefore, according to this configuration, it is possible to generate a virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select a designated position corresponding to the viewpoint position used to generate the virtual viewpoint image 64 from a plurality of candidates.

[0180] In the second embodiment, the other viewpoint position image 92 with the indication position candidate displayed on the display 74 is a two-dimensional image, but the technology of the present disclosure is not limited to this, and may be a stereoscopic image in which a plurality of dot marks 92A or marks replacing these (for example, star marks, triangular marks, or square marks) are visually recognizable via the display 74. The stereoscopic image may be generated based on a parallax image obtained by using a plurality of phase difference pixels, or may be generated based on a vibration image obtained by applying vibration to the HMD camera 76.

[0181] In the second embodiment, the CPU 22A of the information processing device 10 acquires the HMD image 88 with the tentative pointing position from the HMD 68 in the image generation process. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 31, the CPU 22A of the information processing device 10 may acquire a reference image 96 with the tentative pointing position from the user device 12 in the image generation process. The reference image 96 with the tentative pointing position is an image corresponding to the reference image 62 with the pointing position described in the first embodiment, and includes tentative pointing position specifying information 96A. The tentative pointing position specifying information 96A is information corresponding to the pointing position specifying information 62A described in the first embodiment. In other words, the reference image 96 with the tentative pointing position is an image obtained by adding the pointing position specifying information 62A described in the first embodiment to the reference image 60 as the tentative pointing position specifying information 96A.

[0182] In this case, the CPU 22A acquires from the imaging device 30 an image showing a subject including a tentative instruction position (a position corresponding to the instruction position described in the first embodiment above) identified from the tentative instruction position identification information 96A of the reference image 96 with tentative instruction position, as a different viewpoint position image 90 showing the state of the inside of the soccer stadium 36 when observing the inside of the soccer stadium 36 from a viewpoint position different from the current viewpoint position of the user 13.

[0183] The CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and generates a different viewpoint position image 92 with designated position candidates by referring to the acquired three-dimensional area image 32. Specifically, the CPU 22A first identifies a feature point that matches between the reference image 96 with the temporary designated position and the three-dimensional area image 32, thereby identifying the position of the optical axis OA in the soccer stadium 36. Next, the CPU 22A compares the reference image 96 with the temporary designated position with the three-dimensional area image 32, and extracts the coordinates of the temporary designated position in the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. Next, the CPU 22A generates a plurality of designated position candidates by the method described in the second embodiment. Then, the CPU 22A generates a different viewpoint position image 92 with designated position candidates by adding information such as a plurality of designated position candidates to the different viewpoint position image 90. The CPU 22A transmits the different viewpoint position image 92 with designated position candidates to the user device 12 via the transmission / reception device 24. In this case, a different viewpoint position image 92 with pointing position candidates is displayed on the display 18 of the user device 12. Then, when any of the dot marks 92A is selected by the user 13 via the touch panel 20, the position intended by the user 13 as the viewpoint position is determined by the CPU 78A, and pointing position identification information 94 is transmitted to the information processing device 10, similar to the second embodiment.

[0184] In the second embodiment, the finger of the user 13 is detected based on the HMD image 86 obtained by capturing an image of the finger of the user 13 by the HMD camera 76, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 32, the finger of the user 13 may be detected from a plurality of captured images captured by a plurality of imaging devices 30 installed in a soccer stadium 36. The finger of the user 13 may be detected by the CPU 22A or the CPU 78A, etc., based on the HMD image 86 and the plurality of captured images. The method of detecting the finger of the user 13 is not limited to the above, and the finger of the user 13 may be detected by attaching a known device whose position and direction can be specified to the finger of the user 13. In this case, the user 13 can point to the viewpoint position using the finger to which the device is attached, so that the viewpoint position can be determined in the same manner as in the above embodiment. Also, it is not necessarily necessary to detect the finger of the user 13 to determine the viewpoint position. For example, the user 13 may hold a known device whose position and direction can be specified and point to a specific direction, so that the viewpoint position can be determined in the same manner as in the above embodiment.

[0185] In the second embodiment, the designated position on the line from the viewpoint where the soccer stadium 36 is observed to the gaze point, i.e., the designated position on the optical axis OA, is set as the viewpoint position used to generate the virtual viewpoint image 64, but the technology of the present disclosure is not limited to this. For example, a designated position on a line from the viewpoint position 56 (see FIG. 4) to a designated point in the image corresponding to the reference image 60 (for example, on the line of sight of the user 13) may be set as the viewpoint position used to generate the virtual viewpoint image 64.

[0186] In this case, for example, as shown in Fig. 33, another viewpoint position image 98 with pointing position candidates may be displayed on the display 18 of the user device 12. The other viewpoint position image 98 with pointing position candidates is an image equivalent to the other viewpoint position image 92 with pointing position candidates, and differs from the other viewpoint position image 92 with pointing position candidates in that a plurality of dot marks 92A are positioned not on an image indicating the optical axis OA but on an image indicating the line of sight 58A of the user 13. It is not essential to display the image indicating the line of sight 58A, and the image indicating the line of sight 58A does not have to be displayed.

[0187] 4, the line of sight 58A corresponds to the optical axis of the imaging optical system of the imaging device 42. In this case, when the dot mark 92A is touched with the finger of the user 13 via the touch panel 20, a position specified from the coordinates associated with the touched dot mark 92A is set as the viewpoint position used for generating the virtual viewpoint image 64. Therefore, according to this configuration, compared to a case where there is no room to select an indication position corresponding to the viewpoint position used for generating the virtual viewpoint image 64 from a plurality of candidates, it is possible to generate the virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy.

[0188] In the second embodiment, the CPU 22A associates the dot mark 92A with the designated position candidate, but the technology of the present disclosure is not limited to this. For example, the CPU 22A may associate a thumbnail image 100B (see FIG. 36) obtained by reducing the virtual viewpoint image 64 obtained when observing the inside of the soccer stadium 36 with the designated position candidate. Here, how to use the thumbnail image 100B by associating the thumbnail image 100B with the designated position candidate will be described with reference to FIGS. 34 to 38.

[0189] 34 as an example, in the image generation process of the information processing device 10, the CPU 22A generates a different viewpoint position image 100 with designation position candidates (see FIG. 36) by the method described in the second embodiment. The CPU 22A generates a plurality of virtual viewpoint images 64 using each of a plurality of viewpoint positions determined based on each of a plurality of coordinates associated with a plurality of designation position candidates included in the different viewpoint position image 100 with designation position candidates. The CPU 22A then stores each of the generated virtual viewpoint images 64 in the NVM 22B in a state associated with the associated designation position candidate.

[0190] As an example, as shown in FIG. 35, the CPU 22A acquires a plurality of virtual viewpoint images 64 from the NVM 22B, and generates a plurality of thumbnail images 100B (see FIG. 36) corresponding to the plurality of virtual viewpoint images 64. The CPU 22A associates the thumbnail images 100B with each of a plurality of designation position candidates included in the other viewpoint position image 100 with designation position candidates. Then, the CPU 22A calculates the other viewpoint position image with designation position candidates with which the thumbnail images 100B are associated. The other viewpoint position image 100 with the pointing position candidate is transmitted to the HMD 68 via the transmitting / receiving device 24 (see FIG. 1). As a result, the other viewpoint position image 100 with the pointing position candidate is displayed on the display 74 of the HMD 68.

[0191] 36, as an example, a plurality of dot marks 92A are arranged at predetermined intervals along the image showing the optical axis OA in the other viewpoint position image 100 with the indication position candidate. Displaying the image showing the optical axis OA is not essential, and the image showing the optical axis OA does not have to be displayed.

[0192] Furthermore, in the other viewpoint position image 100 with the designation position candidates, thumbnail images 100B linked to each of the multiple dot marks 92A are arranged along the image showing the optical axis OA. Furthermore, a message 100C is added to the other viewpoint position image 100 with the designation position candidates. The message 100C is a message that prompts the user 13 to select a designation position candidate, and in the example shown in Fig. 36, a message "Please select one of the thumbnail images" is displayed as an example of the message 100C.

[0193] As an example, as shown in FIG. 37, when a different viewpoint position image 100 with a candidate pointing position is displayed on the display 74 of the HMD 68 and the fingertip of the user 13 is positioned on one of the thumbnail images 100B, the thumbnail image 100B on which the fingertip is positioned is transmitted by the CPU 78A to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).

[0194] 38 as an example, in the image generation process of the information processing device 10, the CPU 22A acquires a thumbnail image 100B from the HMD 68, and acquires a virtual viewpoint image 64 corresponding to the acquired thumbnail image 100B from the NVM 22B. Then, the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24 (see FIG. 1). As a result, the virtual viewpoint image 64 is displayed on the display 18 of the user device 12.

[0195] 34 to 38, a thumbnail image 100B is associated with each of a plurality of designation position candidates included in the other viewpoint position image 100 with designation position candidates, and a virtual viewpoint image 64 corresponding to the thumbnail image 100B selected by the user 13 is displayed on the display 18 of the user device 12. Therefore, according to this configuration, it is possible to generate a virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select a designation position corresponding to a viewpoint position used to generate the virtual viewpoint image 64 from a plurality of candidates. In addition, the user 13 can predict, through the thumbnail image 100B, what kind of virtual viewpoint image 64 will be provided by the information processing device 10.

[0196] 34 to 38, one of the thumbnail images 100B is selected by the user 13 while the other viewpoint position image 100 with the pointing position candidate is displayed on the display 74 of the HMD 68, but the technology of the present disclosure is not limited to this. For example, the user 13 may select the thumbnail image 100B linked to the pointing position candidate on the line (e.g., on the image showing the line of sight of the user 13) from the viewpoint position 56 (see FIG. 4) toward a specified point (e.g., a tentative pointing position) in the image corresponding to the reference image 60.

[0197] In this case, for example, as shown in Fig. 39, a different viewpoint position image 102 with a pointing position candidate may be displayed on the display 18 of the user device 12. The different viewpoint position image 102 with a pointing position candidate is an image equivalent to the different viewpoint position image 100 with a pointing position candidate, and differs from the different viewpoint position image 100 with a pointing position candidate in that a plurality of thumbnail images 100B are positioned on an image indicating the line of sight 58A of the user 13, not on an image indicating the optical axis OA. The line of sight 58A corresponds to the optical axis of the imaging optical system of the imaging device 42, for example, in the example shown in Fig. 4. The thumbnail image 100B displayed on the display 18 of the user device 12 is an example of a "second reduced image" according to the technology of the present disclosure.

[0198] When the multiple thumbnail images 100B are touched by the finger of the user 13 via the touch panel 20 while the other viewpoint position image 102 with the indication position candidate is displayed on the display 18 of the user device 12, the virtual viewpoint image 64 corresponding to the touched thumbnail image 100B is transmitted to the user device 12 and displayed on the display 18 of the user device 12. Therefore, according to this configuration, it is possible to generate the virtual viewpoint image 64 based on the viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select an indication position corresponding to the viewpoint position used to generate the virtual viewpoint image 64 from multiple candidates. In addition, the user 13 can predict what kind of virtual viewpoint image 64 will be provided from the information processing device 10 through the thumbnail image 100B.

[0199] [Third embodiment] In the above embodiments, cases were described where the indication position can be set without limit within soccer stadium 36, but in this third embodiment, a case will be described where the area in which the indication position can be set is limited. In this third embodiment, the same components as those described in the above embodiments are given the same reference numerals, their description will be omitted, and only the parts that differ from the above embodiments will be described.

[0200] As an example, as shown in Fig. 40, an information processing system 105 according to the third embodiment includes an information processing device 10 and a user device 12. In the information processing device 10, an observation range restriction processing program 104 and a three-dimensional area image 106 with spectator seat information are stored in the NVM 22B. The CPU 22A reads out the observation range restriction processing program 104 from the NVM 22B and executes the observation range restriction processing program 104 on the RAM 22C to perform the observation range restriction process (see Fig. 47).

[0201] The three-dimensional area image 106 with spectator seat information is an image in which spectator seat information 106A (see FIG. 41) is added to the three-dimensional area image 32 described in the first embodiment. A plurality of three-dimensional area images 106 with spectator seat information are stored in the NVM 22B. The plurality of three-dimensional area images 106 with spectator seat information are images defined for each spectator venue, and are used for each spectator venue.

[0202] Spectator seats 36B (see FIG. 2) are divided into classes. The classes are determined according to the purchase price of the spectator ticket, and in the example shown in FIG. 41, the areas of spectator seats 36B are differentiated into classes S seats, A seats, and B seats. As an example, as shown in FIG. 41, the class of spectator seats 36B is reflected as spectator seat information 106A in the three-dimensional area image with spectator seat information.

[0203] Spectating seat information 106A includes class identification information capable of identifying the class of spectating seats 36B, and coordinates capable of identifying each area divided into classes in soccer stadium 36. A class is also assigned to the position of spectating seats 36B where user 13 is watching, and user 13 can only watch in an area of ​​the same class. That is, information processing system 105 allows user 13 to set a viewpoint position for an area in soccer stadium 36 where user 13 can watch, but prohibits user 13 from setting a viewpoint position for other areas.

[0204] Note that the user 13 is an example of an "instruction source" according to the technology of the present disclosure, and the class of the spectator seats 36B is an example of an "attribute" according to the technology of the present disclosure.

[0205] In the information processing system 105, the grade given to the position of the spectator seats 36B where the user 13 is watching the game is identified based on a live view image captured by the user device 12.

[0206] 42, in user device-side processing of the user device 12, the CPU 40A acquires a live view image from the imaging device 42. Then, the CPU 40A transmits the acquired live view image to the information processing device 10 via the transmission / reception device 44 (see FIG. 3).

[0207] As an example, as shown in FIG. 43, in the observation range instruction process of the information processing device 10, the CPU 22A acquires a live view image from the user device 12. The CPU 22A acquires a three-dimensional area image 106 with spectator seat information by referring to the live view image acquired from the user device 12. Specifically, the CPU 22A calculates the degree of coincidence of feature points between the live view image and the three-dimensional area image 32 included in the three-dimensional area image 106 with spectator seat information, and selects and acquires one three-dimensional area image 106 with spectator seat information from the multiple three-dimensional area images 106 with spectator seat information based on the calculated degree of coincidence. That is, the CPU 22A acquires the three-dimensional area image 106 with spectator seat information including the three-dimensional area image 32 with the maximum degree of coincidence of feature points with the live view image. Here, the CPU 22A adds the same class area information to the three-dimensional area image 106 with spectator seat information. The same-rank area information is information (for example, coordinates) capable of identifying a same-rank area 110 (see FIG. 45) that is an area of ​​the same rank as the area in which the user 13 is watching.

[0208] The same-class area information is generated by the CPU 22A based on the live view image and the three-dimensional area image 106 with spectator seat information. The CPU 22A specifies an image having the highest degree of coincidence with the live view image among the three-dimensional area images 32 included in the acquired three-dimensional area image 106 with spectator seat information as a user viewing area image showing the area where the user 13 is watching. The CPU 22A specifies a class corresponding to the user viewing area image by referring to the three-dimensional area image 106 with spectator seat information. The CPU 22A specifies an area of ​​the same class as the specified class, that is, the same-class area 110, by referring to the three-dimensional area image 106 with spectator seat information. The CPU 22A adds the same-class area information, which is information capable of specifying the specified same-class area 110, to the three-dimensional area image 106 with spectator seat information. Then, the CPU 22A transmits the three-dimensional area image 106 with spectator seat information to which the same class area information has been added, to the user device 12 via the transmission / reception device 24 (see FIG. 40).

[0209] As an example, as shown in FIG. 44, in user device-side processing of the user device 12, the CPU 40A acquires a three-dimensional area image 106 with spectator seat information to which the same class area information has been added from the information processing device 10, and acquires a live view image from the imaging device 42. The CPU 40A generates a reference image 108 (see FIG. 45) which is an image based on the three-dimensional area image 106 with spectator seat information to which the same class area information has been added. Specifically, the CPU 40A generates the reference image 108 by using the live view image, with reference to the three-dimensional area image 106 with the same class area information and the spectator seat information. The CPU 40A causes the display 18 to display the reference image 108.

[0210] As an example, as shown in FIG. 45, the reference image 108 includes a target mark 60A and an area of ​​the same rank 110. The area of ​​the same rank 110 is shown in a manner that can be distinguished from other areas in the reference image 108. In the example shown in FIG. 45, the area of ​​the same rank 110 is outlined with a thick line. In the information processing system 105, the user 13 can set the viewpoint position only in the area of ​​the same rank 110. Therefore, even if the user device 12 requests the information processing device 10 to set the viewpoint position in an area other than the area of ​​the same rank 110 in the same manner as in the first embodiment, the information processing device 10 does not respond to the request to set the viewpoint position from the user device 12.

[0211] The three-dimensional area image 106 with spectator seat information to which the same-class area information is added is an example of a "three-dimensional area state image" according to the technology of the present disclosure. Also, the same-class area 110 is an example of an "observation position indication range" according to the technology of the present disclosure.

[0212] Next, the operation of the information processing system 105 will be described.

[0213] First, an example of the flow of user device side processing performed by the CPU 40A of the user device 12 will be described with reference to Fig. 46. The flowchart shown in Fig. 46 differs from the flowchart shown in Fig. 11 in that steps ST250 to ST262 are included before step ST10 in the flowchart shown in Fig. 11. Only steps different from the flowchart shown in Fig. 11 will be described below.

[0214] In the user device-side processing shown in FIG. 46, first, in step ST250, the CPU 40A acquires a live view image from the imaging device 42, and then the user device-side processing proceeds to step ST252.

[0215] In step ST252, the CPU 40A transmits the live view image acquired in step ST250 to the information processing device 10 via the transmission / reception device 44 (see FIG. 3), and thereafter, the user device side processing proceeds to step ST254.

[0216] In step ST254, the CPU 40A judges whether or not the three-dimensional area image 106 with the same class area information and spectator seat information transmitted by executing the process of step ST304 shown in Fig. 47 has been received by the transmission / reception device 44 (see Fig. 3). If the three-dimensional area image 106 with spectator seat information has not been received by the transmission / reception device 44 in step ST254, the judgment is negative, and the user device-side processing proceeds to step ST262. If the three-dimensional area image 106 with spectator seat information has been received by the transmission / reception device 44 in step ST254, the judgment is positive, and the user device-side processing proceeds to step ST256.

[0217] In step ST262, the CPU 40A judges whether or not the user device side processing end condition is satisfied. If the user device side processing end condition is not satisfied in step ST262, the judgment is negative, and the user device side processing proceeds to step ST254. If the user device side processing end condition is satisfied in step ST262, the judgment is positive, and the user device side processing proceeds to step ST256.

[0218] In step ST256, the CPU 40A refers to the three-dimensional area image 106 with the same class area information and spectator seat information received by the transmission / reception device 44 in step ST254, and generates a reference image 108 using the live view image acquired in step ST256, and then the user device side processing proceeds to step ST260.

[0219] In step ST260, CPU 40A causes reference image 108 generated in step ST258 to be displayed on display 18, and thereafter, the user device side processing proceeds to step ST10 (see FIG. 11).

[0220] Next, an example of the flow of observation range restriction processing performed by the CPU 22A of the information processing device 10 will be described with reference to FIG.

[0221] In the observation range restriction process shown in Fig. 47, first, in step ST300, the CPU 22A determines whether or not the live view image transmitted by executing the process of step ST252 shown in Fig. 46 has been received by the transmission / reception device 24 (see Fig. 40). In step ST300, if the live view image has not been received by the transmission / reception device 24, the determination is negative, and the observation range restriction process proceeds to step ST306. In step ST300, if the live view image has been received by the transmission / reception device 24, the determination is positive, and the observation range restriction process proceeds to step ST302.

[0222] In step ST302, the CPU 22A acquires the three-dimensional area image 106 with spectator seat information from the NVM 22B by referring to the live view image received by the transmission / reception device 24 in step ST300. The CPU 22A also generates same-class area information based on the live view image received by the transmission / reception device 24 in step ST300 and the three-dimensional area image 106 with spectator seat information acquired from the NVM 22B. Then, the CPU 22A adds the same-class area information to the generated three-dimensional area image 106 with spectator seat information. After the process of step ST302 is executed, the observation range limiting process proceeds to step ST304.

[0223] In step ST304, the CPU 22A transmits the three-dimensional area image 106 with the same class area information and spectator seat information obtained in step ST304, i.e., the three-dimensional area image 106 with spectator seat information to which the same class area information has been added, to the user device 12, and then the observation range restriction processing proceeds to step ST306.

[0224] In step ST306, the CPU 22A judges whether or not a condition for terminating the observation range restriction process (hereinafter referred to as an "observation range restriction process termination condition") is satisfied. A first example of the observation range restriction process termination condition is that an instruction to terminate the observation range restriction process is given to the information processing device 10 from an administrator of the information processing device 10 or the like. A second example of the observation range restriction process termination condition is that a fourth predetermined time (e.g., 10 hours) has elapsed since the execution of the observation range restriction process was started. A third example of the observation range restriction process termination condition is that the processing capacity of the CPU 22A has decreased to below a reference level.

[0225] In step ST306, if the observation range restriction processing end condition is not satisfied, the determination is negative and the observation range restriction processing proceeds to step ST300. If the observation range restriction processing end condition is satisfied, the determination is positive and the observation range restriction processing ends.

[0226] As described above, in the information processing system 105, the same-class area information is added to the three-dimensional area image 106 with spectator seat information, and the same-class area 110 in which the viewpoint position can be set is determined according to the class of the spectator seats 36B. The same-class area 110 is reflected in the reference image 108 (see FIG. 45). The reference image 108 reflecting the same-class area 110 is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can visually recognize the area in which the viewpoint position can be set.

[0227] Furthermore, in the information processing system 105, the CPU 22A generates a three-dimensional area image 106 with spectator seat information to which the same-class area information is added. The same-class area information is information that can identify the same-class area 110, which is an area of ​​the same class as the area where the user 13 is watching. In other words, the three-dimensional area image 106 with spectator seat information to which the same-class area information is added is an image that can distinguish the same-class area 110 from other areas. Therefore, according to this configuration, the user 13 can easily grasp the areas in which the viewpoint position can be set and the areas in which the viewpoint position cannot be set, compared to the case where an image in which the same-class area 110 cannot be distinguished from other areas is used.

[0228] Moreover, in the information processing system 105, the reference image 108 (see FIG. 45) is an image based on the three-dimensional region image 106 with same-class area information and spectator seat information. That is, the reference image 108 is an image in which the same-class area 110 can be identified, and is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can visually recognize the area in which the viewpoint position can be set.

[0229] In the above third embodiment, an example was described in which the range in which the observation position can be specified according to the class of the spectator seats 36B (in the example shown in FIG. 45, the same class area 110) is determined by the CPU 22A, but the technology of the present disclosure is not limited to this. For example, the range in which the observation position can be specified according to the attributes of the user 13, such as the color of the team the user 13 is cheering for, the color the user 13 likes the most, the gender of the user 13, the age group of the user 13, and the clothes of the user 13, may be determined by the CPU 22A together with the class of the spectator seats 36B or instead of the class of the spectator seats 36B. In this case, for example, a three-dimensional area image with attribute information in which information capable of identifying the attributes of the user 13 is added to the three-dimensional area image 32 may be stored in advance in the NVM 22B. In addition, the range in which the observation position can be specified according to the class of the spectator seats 36B is not limited to the same class area and may be any range. Furthermore, the range in which the observation position can be specified according to the grade of the spectator seats 36B does not need to be divided by the range of the spectator seats, and may be divided, for example, within the soccer field 36A. More specifically, for example, the higher the grade of the spectator seats 36B, the closer the observation position to the goal may be specified.

[0230] In addition, in the above third embodiment, an example was given in which the reference image 108 is displayed on the display 18 of the user device 12, but the technology of the present disclosure is not limited to this, and the reference image 108 may be displayed on the display 74 of the HMD 68.

[0231] In addition, while a soccer stadium 36 has been given as an example in each of the above embodiments, the technology of the present disclosure is not limited to this, and the venue may be any location where multiple imaging devices 30 can be installed, such as a baseball field, rugby field, curling court, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater venue.

[0232] In addition, in each of the above embodiments, the computer 22 is exemplified, but the technology of the present disclosure is not limited thereto. For example, a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 22. Also, a combination of a hardware configuration and a software configuration may be used instead of the computer 22. The same applies to the computers 40 and 78.

[0233] In the above embodiment, the image generation processing program 34 and the observation range restriction processing program 104 (hereinafter, referred to as "programs" unless it is necessary to distinguish between them in the following description) are stored in the NVM 22B, but the technology of the present disclosure is not limited to this, and as an example, as shown in FIG. 48, the programs may be stored in any portable storage medium 200 such as an SSD or USB memory, which is a non-transitory storage medium. In this case, the program stored in the storage medium 200 is installed in the computer 22, and the CPU 22A performs image generation processing and observation range restriction processing (hereinafter, referred to as "programs" unless it is necessary to distinguish between them in the following description) in accordance with the program. If no special processing is required, this is called "special processing."

[0234] Moreover, the program may be stored in a storage unit of another computer or a server device connected to the computer 22 via a communication network (not shown), and the program may be downloaded to the information processing device 10 in response to a request from the information processing device 10. In this case, the specific process based on the downloaded program is executed by the CPU 22A of the computer 22.

[0235] In addition, in each of the above embodiments, the CPU 22A is exemplified, but the technology of the present disclosure is not limited to this, and a GPU may be adopted. Moreover, multiple CPUs may be adopted instead of the CPU 22A. In other words, a specific process may be executed by one processor, or multiple processors that are physically separated.

[0236] As the hardware resource for executing the specific process, various processors as shown below can be used. As the processor, for example, as described above, there is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process according to software, i.e., a program. In addition, as another processor, there is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a dedicated process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0237] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0238] As an example of a configuration using one processor, first, there is a configuration in which one processor is configured by combining one or more CPUs and software, as typified by computers such as client and server, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute a specific process in a single IC chip, as typified by SoC. In this way, the specific process is realized using one or more of the above various processors as hardware resources.

[0239] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0240] Furthermore, the specific processing described above is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed without departing from the spirit of the present invention.

[0241] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0242] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0243] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor; A memory included in or connected to the processor; The processor, acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indication position indicated within the three-dimensional area to be observed or indicated within a reference image showing the state of the three-dimensional area when the three-dimensional area is observed from a reference position; the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the viewpoint position is a position corresponding to the position of the imaging device, The subject image includes a second mark capable of identifying the designated position designated in the reference image. Information processing device.

2. The processor derives the coordinates based on an observation manner in which the three-dimensional area is observed and the indicated position. The information processing device according to claim 1 .

3. The observation mode is determined according to an observation position at which the three-dimensional area is observed. The information processing device according to claim 2 .

4. The processor determines an observation position designation range within which the observation position can be designated according to an attribute of a designation source. The information processing device according to claim 3 .

5. The processor acquires a three-dimensional area interior appearance image showing an appearance within the three-dimensional area when the three-dimensional area is observed in the observation manner, The three-dimensional area status image is an image in which the observation position designation range in the three-dimensional area and a range other than the observation position designation range are displayed in a distinguishable manner. The information processing device according to claim 4.

6. The reference image is an image based on the three-dimensional region interior aspect image. The information processing device according to claim 5 .

7. The processor derives the coordinates based on a correspondence relationship between an image showing a state of the three-dimensional area when the three-dimensional area is observed in the observation state and a three-dimensional area image showing the three-dimensional area and whose position can be specified by the coordinates. The information processing device according to any one of claims 2 to 6.

8. The reference image is a virtual viewpoint image generated based on a plurality of images obtained by capturing images of the three-dimensional area using a plurality of imaging devices, or an image based on captured images obtained by capturing images of the three-dimensional area. The information processing device according to claim 1 .

9. The designated position designated in the reference image is a specific position in the virtual viewpoint image or the captured image. The information processing device according to claim 8.

10. The reference image is an image including a first mark that can identify the indicated position within the reference image. The information processing device according to any one of claims 1 to 9.

11. When an object image showing an object present in the three-dimensional area when the three-dimensional area is observed from a position whose distance from the indicated position is within a threshold range or less is stored in a storage area, the processor acquires the object image instead of the subject image. The information processing device according to any one of claims 1 to 10.

12. The coordinates of a specific region within the three-dimensional region are coordinates that indicate a position higher than the actual position of the specific region within the three-dimensional region. The information processing device according to claim 1 .

13. the designated position designated within the three-dimensional region is a designated position on a first line extending from a viewpoint at which the three-dimensional region is observed toward a gaze point, The designated position designated in the reference image is a designated position on a second line extending from the reference position to a specified point in the reference image. The information processing device according to claim 1 .

14. the designated position designated within the three-dimensional region is a position selected from at least one first candidate position; the designated position designated in the reference image is a position selected from at least one second candidate position; the processor associates, with the at least one first candidate position, a first reduced image obtained by reducing the object image when the three-dimensional region is observed from the first candidate position; A second reduced image obtained by reducing the subject image when the three-dimensional area is observed from the second candidate position is associated with the at least one second candidate position. The information processing device according to any one of claims 1 to 13.

15. The processor detects the pointing position based on a designated area image showing an area designated within the three-dimensional area. The information processing device according to any one of claims 1 to 14.

16. The subject image is a virtual viewpoint image generated based on a plurality of images obtained by capturing images of the three-dimensional area using a plurality of imaging devices. The information processing device according to any one of claims 1 to 15.

17. acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indication position indicated within the three-dimensional area to be observed, or indicated within a reference image showing the state of the three-dimensional area when the three-dimensional area is observed from a reference position; the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the viewpoint position is a position corresponding to the position of the imaging device, The subject image includes a second mark capable of identifying the designated position designated in the reference image. Information processing methods.

18. A program for causing a computer to execute a process, acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indication position indicated within the three-dimensional area to be observed, or indicated within a reference image showing the state of the three-dimensional area when the three-dimensional area is observed from a reference position; the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the viewpoint position is a position corresponding to the position of the imaging device, The subject image includes a second mark capable of identifying the designated position designated in the reference image. program.

Images