Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program which enable a user to observe a state inside a three-dimensional region from various positions.SOLUTION: An information processing apparatus includes a processor, and a memory built in or connected to the processor. The processor acquires a subject image showing a subject present inside a three-dimensional region when the three-dimensional region is observed from a viewpoint position determined based on coordinates inside the three-dimensional region corresponding to an indication position that is indicated inside the three-dimensional region to be observed or that is indicated inside a reference image showing a state of the inside of the three-dimensional region in a case in which the inside of the three-dimensional region is observed from a reference position.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] JP 2019-133309 A discloses a program for causing a computer to execute the following steps: setting a virtual space to provide a user with a virtual experience; setting a plurality of movement areas in the virtual space; setting a virtual viewpoint in the virtual space; indicating a predetermined movement area among the plurality of movement areas in accordance with the movement of a part of the user's body; moving the virtual viewpoint to the predetermined movement area if the distance between the virtual viewpoint and the predetermined movement area is equal to or less than a first threshold; and not moving the virtual viewpoint to the predetermined movement area if the distance between the virtual viewpoint and the predetermined movement area exceeds the first threshold. Summary of the Invention

[0003] One embodiment of the technique of the present disclosure provides an information processing device, an information processing method, and a program that allow a user to observe a state in a three-dimensional area from various positions. [Means for solving the problem]

[0004] A first aspect of the technology of the present disclosure is an information processing device that includes a processor and a memory built into or connected to the processor, and that acquires a subject image showing a subject present in a three-dimensional area when the processor observes the three-dimensional area from a viewpoint position determined based on coordinates within the three-dimensional area that correspond to an indicated position within the three-dimensional area being observed, or an indicated position within a reference image that shows the appearance of the three-dimensional area when observed from a reference position.

[0005] A second aspect of the technique of the present disclosure is the information processing device according to the first aspect, in which the processor derives coordinates based on an observation mode in which the three-dimensional area is observed and a pointed position.

[0006] A third aspect according to the technique of the present disclosure is the information processing device according to the second aspect, in which the observation mode is determined according to an observation position at which the three-dimensional region is observed.

[0007] A fourth aspect of the technology of the present disclosure is an information processing device according to the third aspect, in which the processor determines an observation position designation range in which an observation position can be designated according to an attribute of the designation source.

[0008] A fifth aspect of the technology of the present disclosure is an information processing device according to the fourth aspect, in which a processor acquires an image of the state within the three-dimensional area showing the state within the three-dimensional area when the three-dimensional area is observed in an observation state, and the image of the state within the three-dimensional area is an image shown in a manner in which the observation position indication range within the three-dimensional area and the range outside the observation position indication range can be distinguished.

[0009] A sixth aspect of the technique of the present disclosure is the information processing device according to the fifth aspect, in which the reference image is an image based on the three-dimensional area interior state image.

[0010] A seventh aspect of the technology of the present disclosure is an information processing device according to any one of the second to sixth aspects, in which a processor derives coordinates based on a correspondence between an image showing the state of the three-dimensional area when the three-dimensional area is observed in an observation state and a three-dimensional area image showing the three-dimensional area and whose position can be identified by coordinates.

[0011] An eighth aspect of the technology of the present disclosure is an information processing device according to any one of the first to seventh aspects, in which the reference image is a virtual viewpoint image generated based on multiple images obtained by capturing images of a three-dimensional area using multiple imaging devices, or an image based on captured images obtained by capturing images of a three-dimensional area.

[0012] A ninth aspect according to the technology of the present disclosure is the information processing device according to the eighth aspect, in which the designated position designated in the reference image is a specific position in the virtual viewpoint image or the captured image.

[0013] A tenth aspect of the technology of the present disclosure is an information processing device according to any one of the first to ninth aspects, in which the reference image is an image including a first mark that can identify an indication position within the reference image.

[0014] An eleventh aspect of the technology of the present disclosure is an information processing device according to any one of the first to tenth aspects, in which the subject image includes a second mark that can identify an indicated position indicated within the reference image.

[0015] A twelfth aspect of the technology of the present disclosure is an information processing device according to any one of the first to eleventh aspects, in which, when an object image showing an object present in a three-dimensional region is stored in a storage area when the processor observes the three-dimensional region from a position whose distance from the indicated position is within a threshold range, the processor acquires the object image instead of the subject image.

[0016] A thirteenth aspect of the technology of the present disclosure is an information processing device according to any one of the first to twelfth aspects, in which coordinates for a specific area within a three-dimensional area are coordinates indicating a position higher than the actual position of the specific area within the three-dimensional area.

[0017] A fourteenth aspect of the technology of the present disclosure is an information processing device according to any one of the first to thirteenth aspects, in which the indicated position indicated within the three-dimensional region is an indicated position on a first line extending from a viewpoint from which the three-dimensional region is observed to a point of gaze, and the indicated position indicated within the reference image is an indicated position on a second line extending from the reference position to a specified point within the reference image.

[0018] A fifteenth aspect of the technology of the present disclosure is an information processing device according to any one of the first to fourteenth aspects, in which a designated position designated within a three-dimensional area is a position selected from at least one first candidate position, a designated position designated within a reference image is a position selected from at least one second candidate position, and the processor associates, with at least one first candidate position, a first reduced image obtained by reducing the image of the subject when observing the three-dimensional area from the first candidate position, and associates, with at least one second candidate position, a second reduced image obtained by reducing the image of the subject when observing the three-dimensional area from the second candidate position.

[0019] A 16th aspect of the technology of the present disclosure is an information processing device according to any one of the first to fifteenth aspects, in which a processor detects a pointed position based on a designated area image showing an area designated within a three-dimensional area.

[0020] A 17th aspect of the technology of the present disclosure is an information processing device according to any one of the first to sixteenth aspects, in which the subject image is a virtual viewpoint image generated based on multiple images obtained by capturing images of a three-dimensional area using multiple imaging devices.

[0021] An 18th aspect of the technology of the present disclosure is an information processing method that includes acquiring a subject image that shows a subject present in a three-dimensional area when the three-dimensional area is observed from a viewpoint position that is determined based on coordinates within the three-dimensional area that correspond to an indicated position within the three-dimensional area that is the object of observation, or an indicated position within a reference image that shows the appearance of the three-dimensional area when the three-dimensional area is observed from a reference position.

[0022] A 19th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a subject image showing a subject present in a three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indicated position within the three-dimensional area being observed, or indicated within a reference image showing the appearance of the three-dimensional area when observed from a reference position. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 10 is a conceptual diagram showing an example of the configuration of a three-dimensional area image. [Figure 3] FIG. 2 is a block diagram illustrating an example of the hardware configuration of an electrical system of a user device. [Figure 4] FIG. 1 is a schematic perspective view showing an example of an image of the inside of a soccer stadium captured by an imaging device of a smart device. [Figure 5] FIG. 4 is a conceptual diagram showing an example of the contents of a user device-side process according to the first embodiment. [Figure 6] FIG. 4 is a conceptual diagram showing an example of the contents of a user device-side process according to the first embodiment. [Figure 7] FIG. 4 is a conceptual diagram showing an example of the contents of a user device-side process according to the first embodiment. [Figure 8] FIG. 4 is a conceptual diagram showing an example of the contents of a user device-side process according to the first embodiment. [Figure 9] FIG. 2 is a conceptual diagram showing an example of the contents of image generation processing according to the first embodiment. [Figure 10] FIG. 2 is a conceptual diagram showing an example of the contents of image generation processing according to the first embodiment. [Figure 11] 10 is a flowchart showing an example of the flow of a user device-side process according to the first embodiment. [Figure 12] 5 is a flowchart showing an example of the flow of image generation processing according to the first embodiment. [Figure 13] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the first embodiment. [Figure 14] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the first embodiment. [Figure 15] FIG. 1 is a conceptual diagram showing an example of the configuration of an HMD. [Figure 16] FIG. 11 is a conceptual diagram showing an example of the content of HMD-side processing according to the second embodiment. [Figure 17] FIG. 2 is a conceptual diagram showing an example of a display mode of the display of the HMD. [Figure 18] FIG. 10 is a conceptual diagram used to explain a method for setting a tentative designated position. [Figure 19] FIG. 11 is a conceptual diagram showing an example of the content of HMD-side processing according to the second embodiment. [Figure 20] FIG. 10 is a conceptual diagram showing an example of the configuration of an HMD image with a temporary designated position. [Figure 21] FIG. 10 is a conceptual diagram showing an example of the contents of an image generation process according to the second embodiment. [Figure 22] FIG. 10 is a conceptual diagram showing an example of the configuration of a different viewpoint position image with a designated position candidate. [Figure 23] FIG. 11 is a conceptual diagram showing an example of the content of HMD-side processing according to the second embodiment. [Figure 24] FIG. 2 is a conceptual diagram showing an example of the display content of the HMD display. [Figure 25] FIG. 11 is a conceptual diagram showing an example of the content of HMD-side processing according to the second embodiment. [Figure 26] FIG. 10 is a conceptual diagram showing an example of the contents of an image generation process according to the second embodiment. [Figure 27] 10 is a flowchart showing an example of the flow of a user device-side process according to the second embodiment. [Figure 28] 10 is a flowchart showing an example of the flow of HMD-side processing according to the second embodiment. [Figure 29] This is a continuation of the flowchart shown in Figure 28. [Figure 30]10 is a flowchart showing an example of the flow of image generation processing according to the second embodiment. [Figure 31] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the second embodiment. [Figure 32] FIG. 1 is a schematic perspective view showing an example of an aspect in which images of a user's fingers are captured by a plurality of imaging devices. [Figure 33] FIG. 10 is a conceptual diagram showing an example of a mode in which a different viewpoint position image with a pointing position candidate is displayed on the display of a user device. [Figure 34] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the second embodiment. [Figure 35] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the second embodiment. [Figure 36] FIG. 10 is a conceptual diagram showing an example of the configuration of a different viewpoint position image with a designated position candidate. [Figure 37] FIG. 10 is a conceptual diagram showing a modified example of the content of the HMD-side processing according to the second embodiment. [Figure 38] FIG. 10 is a conceptual diagram showing a modified example of the content of the image generation processing according to the second embodiment. [Figure 39] FIG. 10 is a conceptual diagram showing an example of a mode in which a different viewpoint position image with a pointing position candidate is displayed on the display of a user device. [Figure 40] FIG. 10 is a conceptual diagram illustrating an example of the configuration of an information processing system according to a third embodiment. [Figure 41] FIG. 10 is a conceptual diagram showing an example of the configuration of a three-dimensional area image with spectator seat information. [Figure 42] FIG. 11 is a conceptual diagram showing an example of the contents of a user device-side process according to the third embodiment. [Figure 43] FIG. 11 is a conceptual diagram showing an example of the contents of an image generation process according to the third embodiment. [Figure 44] FIG. 11 is a conceptual diagram showing an example of the contents of a user device-side process according to the third embodiment. [Figure 45] FIG. 10 is a conceptual diagram showing an example of a reference image being displayed on the display of a user device. [Figure 46]11 is a flowchart showing an example of the flow of a user device-side process according to the third embodiment. [Figure 47] 11 is a flowchart showing an example of the flow of observation range restriction processing according to the third embodiment. [Figure 48] FIG. 1 is a conceptual diagram showing an example of how a program stored in a storage medium is installed in a computer of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of an information processing device, an information processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-Volatile Memory". RAM is an abbreviation for "Random Access Memory". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". GPU is an abbreviation for "Graphics Processing Unit". LAN is an abbreviation for "Local Area Network". 3D is an abbreviation for "3 Dimensions". USB is an abbreviation for "Universal Serial Bus". "HMD" is an abbreviation for "Head Mounted Display". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)". TDM is an abbreviation for "Time-Division Multiplexing". HMD is an abbreviation for "Head Mounted Display".For ease of explanation, a CPU is exemplified below as an example of a "processor" according to the technology of the present disclosure, but the "processor" according to the technology of the present disclosure may be a combination of multiple processing devices such as a CPU and a GPU. When a combination of a CPU and a GPU is applied as an example of a "processor" according to the technology of the present disclosure, the GPU operates under the control of the CPU and is responsible for executing image processing.

[0027] In the following description, "match" refers to a match that includes not only a perfect match, but also an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs, and that does not contradict the spirit of the technology of the present disclosure.

[0028] [First embodiment] As an example, as shown in FIG. 1, an information processing system 2 includes an information processing device 10 and a user device 12.

[0029] In the first embodiment, a server is applied as an example of the information processing device 10. However, this is merely an example, and the information processing device 10 may be a personal computer, multiple personal computers, multiple servers, or a device that combines a personal computer and a server.

[0030] In the first embodiment, a smartphone is used as an example of the user device 12. However, the smartphone is merely an example, and the user device 12 may be, for example, a personal computer, a tablet terminal, an HMD, or another portable multi-function terminal.

[0031] In the first embodiment, the information processing device 10 and the user device 12 are communicatively connected via, for example, a base station (not shown). Communication standards used in the base station include wireless communication standards including the 5G standard, the LTE standard, etc., wireless communication standards including the WiFi (802.11) standard and / or the Bluetooth (registered trademark) standard, and wired communication standards including the TDM standard and / or the Ethernet (registered trademark) standard.

[0032] The information processing device 10 acquires an image and transmits the acquired image to the user device 12. Here, the image refers to, for example, a captured image obtained by capturing an image, an image generated based on the captured image, etc. An example of an image generated based on a captured image is a virtual viewpoint image.

[0033] The user device 12 is used by a user 13. The user device 12 includes a touch panel display 16. The touch panel display 16 is realized by a display 18 and a touch panel 20. An example of the display 18 is an EL display (e.g., an organic EL display or an inorganic EL display). Note that the display is not limited to an EL display, and may be another type of display such as a liquid crystal display.

[0034] The touch panel display 16 is formed by overlaying the touch panel 20 on the display area of ​​the display 18, or by using an in-cell type in which the touch panel function is built into the display 18. Note that the in-cell type is merely an example, and the display may also be an out-cell type or an on-cell type.

[0035] The user device 12 executes processing (e.g., user device processing, which will be described later) in accordance with instructions received from a user via the touch panel 20, etc. For example, the user device 12 exchanges various information with the information processing device 10 in accordance with instructions received from a user via the touch panel 20, etc.

[0036] The user device 12 receives the image transmitted from the information processing device 10 and displays the received image on the display 18. The user 13 views the image displayed on the display 18.

[0037] The information processing device 10 includes a computer 22, a transmission / reception device 24, a communication I / F 26, and a bus 28. The computer 22 includes a CPU 22A, an NVM 22B, and a RAM 22C, and the CPU 22A, the NVM 22B, and the RAM 22C are connected via the bus 28. In the example shown in Fig. 1, for convenience of illustration, a single bus is shown as the bus 28, but multiple buses may be used. The bus 28 may also include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, etc.

[0038] The CPU 22A is an example of a "processor" according to the technology of the present disclosure. The CPU 22A controls the entire information processing device 10. The NVM 22B stores various parameters, various programs, etc. Examples of the NVM 22B include an EEPROM, an SSD, and / or an HDD. The RAM 22C is an example of a "memory" according to the technology of the present disclosure. Various pieces of information are temporarily stored in the RAM 22C. The RAM 22C is used as a work memory by the CPU 22A.

[0039] The transceiver 24 is connected to the bus 28. The transceiver 24 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the user device 12 via a base station (not shown) under the control of the CPU 22A. In other words, the CPU 22A transmits and receives various types of information to and from the user device 12 via the transceiver 24.

[0040] The communication I / F 26 is realized by, for example, a device having an FPGA. The communication I / F 26 is connected to a plurality of image capture devices 30 via LAN cables (not shown). The image capture devices 30 are image capture devices having CMOS image sensors and equipped with optical zoom functions and / or digital zoom functions. Note that other types of image sensors, such as CCD image sensors, may be used instead of the CMOS image sensors.

[0041] The multiple imaging devices 30 are installed in a soccer stadium 36 (see FIG. 2) and capture images of subjects within the soccer stadium 36. The captured images obtained by capturing images of subjects by the imaging devices 30 are used, for example, to generate a virtual viewpoint image. For this reason, the multiple imaging devices 30 are installed in different locations within the soccer stadium 36 (see FIG. 2), i.e., locations that can obtain multiple captured images from which a virtual viewpoint image can be generated.

[0042] The communication I / F 26 is connected to the bus 28 and controls the exchange of various information between the CPU 22A and the multiple imaging devices 30. For example, the communication I / F 26 controls the multiple imaging devices 30 in accordance with a request from the CPU 22A. The communication I / F 26 outputs captured images (hereinafter also simply referred to as "captured images") obtained by capturing images using each of the multiple imaging devices 30 to the CPU 22A. Note that although the communication I / F 26 is exemplified here as a wired communication I / F, it may also be a wireless communication I / F such as a high-speed wireless LAN.

[0043] The NVM 22B stores a three-dimensional area image 32 and an image generation processing program 34. As will be described in detail later, the three-dimensional area image 32 is a three-dimensional image that shows the appearance of a three-dimensional area, and the three-dimensional area image 32 is assigned coordinates that can identify a position within the three-dimensional area.

[0044] The image generation processing program 34 is an example of a "program" according to the technology of the present disclosure. The CPU 22A reads the image generation processing program 34 from the NVM 22B and executes the image generation processing program 34 on the RAM 22C to perform image generation processing (see FIG. 12).

[0045] As an example, as shown in FIG. 2, the three-dimensional area image 32 is a three-dimensional image showing a soccer stadium 36. The soccer stadium 36 is an example of a "three-dimensional area" according to the technology of the present disclosure. The soccer stadium 36 is a three-dimensional area including a soccer field 36A and spectator seats 36B constructed to surround the soccer field 36A, and is the object of observation for the user 13. The example shown in FIG. 2 shows an aspect in which the observer, i.e., the user 13, is observing the inside of the soccer stadium 36 from the spectator seats 36B.

[0046] The three-dimensional area image 32 is provided with coordinates that can identify a position within the soccer stadium 36. Here, as an example of coordinates that can identify a position within the soccer stadium 36, three-dimensional coordinates that can identify a position within a rectangular parallelepiped 38 that surrounds the soccer stadium 36, with one vertex of the rectangular parallelepiped 38 as the origin, are applied.

[0047] The coordinates relating to the position of the soccer field 36A shown in the three-dimensional area image 32 are coordinates indicating a position higher than the actual position of the soccer field 36A. Here, the coordinates relating to the position of the soccer field 36A refer to the coordinates assigned to the three-dimensional area image 32 that are assigned to the position of the soccer field 36A. Furthermore, here, the coordinates indicating a position higher than the actual position are, for example, coordinates indicating a position higher than the actual position by a distance corresponding to the average height of a typical adult. The soccer field 36A is an example of a "specific area" according to the technology of the present disclosure.

[0048] As an example, as shown in FIG. 3, the user device 12 includes a display 18, a computer 40, an imaging device 42, a transceiver device 44, a speaker 46, a microphone 48, an accepting device 50, and a bus 52.

[0049] The computer 40 includes a CPU 40A, an NVM 40B, and a RAM 40C, which are connected to each other via a bus 52. In the example shown in Fig. 3, for convenience of illustration, a single bus is shown as the bus 52, but multiple buses may be used. The bus 52 may also include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, and the like.

[0050] The CPU 40A controls the entire user device 12. The NVM 40B stores various parameters and programs. An example of the NVM 40B is an EEPROM. The RAM 40C temporarily stores various pieces of information. The RAM 40C is used as a work memory by the CPU 40A.

[0051] The imaging device 42 is an imaging device having a CMOS image sensor and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of the CMOS image sensor. The imaging device 42 is connected to a bus 52, and the CPU 40A controls the imaging device 42. The captured image obtained by imaging the imaging device 42 is acquired by the CPU 40A via the bus 52.

[0052] The transmitting / receiving device 44 is connected to the bus 52. The transmitting / receiving device 44 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various information to and from the information processing device 10 via a base station (not shown) under the control of the CPU 40A. That is, the CPU 40A transmits and receives various information to and from the information processing device 10 via the transmitting / receiving device 44.

[0053] The speaker 46 converts the electrical signal into sound. The speaker 46 is connected to the bus 52. The speaker 46 receives the electrical signal output from the CPU 40A via the bus 52, converts the received electrical signal into sound, and outputs the sound obtained by converting the electrical signal to the outside of the user device 12.

[0054] The microphone 48 converts the collected sound into an electrical signal. The microphone 48 is connected to a bus 52. The electrical signal obtained by converting the sound collected by the microphone 48 is acquired by the CPU 40A via the bus 52.

[0055] The receiving device 50 receives instructions from the user 13, etc. Examples of the receiving device 50 include a touch panel 20 and hard keys (not shown). The receiving device 50 is connected to a bus 52, and the instructions received by the receiving device 50 are acquired by the CPU 40A.

[0056] The NVM 40B stores a user device side processing program 54. The CPU 40A reads the user device side processing program 54 from the NVM 40B and executes the user device side processing program 54 on the RAM 40C to perform user device side processing (see FIG. 11).

[0057] 4, the observation mode in which the user 13 observes the inside of the soccer stadium 36 (hereinafter also simply referred to as the "observation mode") is defined by the viewpoint position 56, line of sight direction 58, and viewing angle θ of the user 13. The viewpoint position 56 corresponds to the position in real space from which the user 13 observes the inside of the soccer stadium 36, and is an example of the "reference position" and "observation position" according to the technology of the present disclosure.

[0058] The viewpoint position 56 is a position corresponding to the position of the imaging device 42 mounted on the user device 12, the line of sight direction 58 is a direction corresponding to the direction of the optical axis of an imaging optical system (not shown) included in the imaging device 42, and the viewing angle θ is an angle corresponding to the angle of view of the imaging device 42. In the information processing system 2, the area observed by the user 13 in real space (actual space) is identified from a captured image obtained by capturing an image of the inside of the soccer stadium 36 by the imaging device 42. Then, in the information processing system 2, a viewpoint position different from the viewpoint position 56 is set using the captured image, and the user 13 views a subject image showing subjects present in the soccer stadium 36 when observing the inside of the soccer stadium 36 from the set viewpoint position. To achieve such viewing, the information processing system according to the first embodiment performs the following user device-side processing and image generation processing.

[0059] As an example, as shown in FIG. 5, in user device-side processing, the CPU 40A acquires a live view image obtained by capturing an image using the imaging device 42. The live view image is an image showing a designated area within the soccer stadium 36. Here, the designated area within the soccer stadium 36 refers to an area defined by, for example, a viewpoint position 56, a line of sight direction 58, and a viewing angle θ. The live view image is an example of an "image showing the state of a three-dimensional area when the three-dimensional area is observed in an observation state" according to the technology of the present disclosure. The CPU 40A generates a reference image 60 using the acquired live view image.

[0060] The reference image 60 is an example of a “designated area image” according to the technology of the present disclosure. The reference image 60 is an image showing the state of the interior of the soccer stadium 36 when the interior of the soccer stadium 36 is observed from the viewpoint position 56 (see FIG. 4 ). The reference image 60 is an image based on a live-view image. In the example shown in FIG. 5 , an image in which a cross-shaped target mark 60A is superimposed on the live-view image is shown as an example of the reference image 60. The target mark 60A is a mark that moves within the reference image 60 according to instructions given by the user 13, and indicates the position designated by the user 13 as the viewpoint position from which the user 13 observes the interior of the soccer stadium 36. In other words, the target mark 60A is a mark that can identify the position designated by the user 13 within the reference image 60. Note that the position of the target mark 60A within the reference image 60, i.e., the position of the target mark 60A superimposed on the live-view image, is an example of a “designated position,” a “specific position within a captured image,” and a “first mark” according to the technology of the present disclosure.

[0061] The home position of the target mark 60A is the center of the reference image 60. In the example shown in Fig. 5, the center of the target mark 60A is located at the center of the reference image 60. The CPU 40A causes the display 18 to display the reference image 60.

[0062] As an example, as shown in FIG. 6, in the user device side processing, the CPU 40A When the touch panel 20 receives a change instruction to change the position of the target mark 60A while the image 60 is displayed on the display 18, the position of the target mark 60A in the reference image 60 is changed in accordance with the change instruction. The change instruction is a swipe performed on the touch panel 20 on the target mark 60A displayed on the display 18. That is, the user 13 touches the target mark 60A via the touch panel 20 and slides the touched position on the touch panel 20 to indicate the new position of the target mark 60A. The CPU 40A updates the reference image 60 displayed on the display 18 to the reference image 60 in which the position of the target mark 60A has been changed. Note that when the reference image 60 is a live view image, instead of touching and changing the position of the target mark 60A on the display 18, the user 13 may move the user device 12 to move the position of the target mark 60A relative to the reference image 60.

[0063] As an example, as shown in FIG. 7 , in the user device-side processing, when a confirmation instruction, which is an instruction to confirm the position of a target mark 60A within the reference image 60, is received by the touch panel 20 while the reference image 60 is displayed on the display 18, the CPU 40A generates a reference image 62 with a pointing position attached. The reference image 62 with a pointing position attached is an image in which pointing position identification information 62A is added to the reference image 60. The pointing position identification information 62A refers to information that can identify the position indicated by the user 13 as the viewpoint position from which the user 13 observes the inside of the soccer stadium 36, i.e., information that can identify the position of the target mark 60A within the reference image 60 (for example, information that can identify the position of the pixel corresponding to the center of the target mark 60A within the reference image 60). The CPU 40A transmits the reference image 62 with a pointing position attached to the information processing device 10 via the transmission / reception device 44 (see FIG. 3 ).

[0064] As an example, as shown in FIG. 8 , in the user device-side processing, the CPU 40A acquires a virtual viewpoint image 64 generated by the information processing device 10. The virtual viewpoint image 64 is a moving image. However, this is merely an example, and the virtual viewpoint image 64 may be a still image. The CPU 40A displays the acquired virtual viewpoint image 64 on the display 18. Here, the CPU 40A does not simply display the virtual viewpoint image 64 on the display 18, but displays it on the display 18 as a new reference image 60.

[0065] The new reference image 60 is an image based on the virtual viewpoint image 64. That is, the new reference image 60 here refers to an image in which a target mark 60A is superimposed on the virtual viewpoint image 64. In the example shown in FIG. 8, an image in which a target mark 60A is superimposed on the virtual viewpoint image 64 so that the target mark 60A is positioned at the center of the virtual viewpoint image 64 is shown as the new reference image 60 displayed on the display 18. Furthermore, as will be described in detail later, the virtual viewpoint image 64 includes a mark 61 (see FIG. 10) that can identify a position within the virtual viewpoint image 64 that corresponds to the designated position used to generate the virtual viewpoint image 64, and the mark 61 is also included in the new reference image 60.

[0066] The new reference image 60 is an example of a "designated area image" according to the technology of the present disclosure. The virtual viewpoint image 64 is an example of an "image showing the state of the interior of a three-dimensional area when the interior of the three-dimensional area is observed in an observation state" according to the technology of the present disclosure. The position of the target mark 60A superimposed on the virtual viewpoint image 64 is an example of an "indicated position" and a "specific position within the virtual viewpoint image" according to the technology of the present disclosure.

[0067] 9, in the image generation process, the CPU 22A acquires a reference image 62 with a designated position from the user device 12. The CPU 22A also acquires a three-dimensional area image 32 from the NVM 22B.

[0068] In the image generation process, the CPU 22A acquires a subject image showing a subject present within the soccer stadium 36 when observing the soccer stadium 36 from a viewpoint position determined based on coordinates within the soccer stadium 36 corresponding to the position of the target mark 60A in the reference image 60.

[0069] 10 as an example, the CPU 22A first compares the reference image 62 with the designated position acquired from the user device 12 with the three-dimensional area image 32 acquired from the NVM 22B to identify matching feature points between the reference image 62 with the designated position and the three-dimensional area image 32. This identifies which pixel in the reference image 62 with the designated position corresponds to which pixel in the three-dimensional area image 32.

[0070] The CPU 22A derives coordinates within the soccer stadium 36 corresponding to the indicated position from the three-dimensional area image 32 based on the observation manner in which the user 13 observes the soccer stadium 36 (in the example shown in Figure 4, the viewpoint position 56, the line of sight direction 58, and the field of view angle θ) and the position of the target mark 60A within the reference image 60 (hereinafter, in this first embodiment, this position will also be simply referred to as the "indicated position").

[0071] The observation manner in which the user 13 observes the inside of the soccer stadium 36 is determined according to the viewpoint position 56, and changes according to the displacement of the viewpoint position 56. Since the observation manner in which the user 13 observes the inside of the soccer stadium 36 is represented by the reference image 62 with the designated position, the CPU 22A derives coordinates in the soccer stadium 36 corresponding to the designated position from the three-dimensional area image 32 based on the correspondence between the reference image 62 with the designated position and the three-dimensional area image 32.

[0072] Specifically, the CPU 22A uses the comparison result between the reference image 62 with the indicated position and the three-dimensional area image 32 (for example, the comparison result as to which pixel in the reference image 62 with the indicated position corresponds to which pixel in the three-dimensional area image 32) to extract the coordinates of the position corresponding to the indicated position from the three-dimensional area image 32.

[0073] The CPU 22A generates a virtual viewpoint image 64 using a viewpoint position determined based on coordinates extracted from the three-dimensional area image 32. The viewpoint position determined based on coordinates extracted from the three-dimensional area image 32 refers to, for example, a position within the soccer stadium 36 that is identified by the coordinates extracted from the three-dimensional area image 32.

[0074] The virtual viewpoint image 64 is a type of subject image that shows subjects present in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position determined based on coordinates extracted from the three-dimensional area image 32. In this case, the observation mode (the observation mode in which the user 13 virtually observes the inside of the soccer stadium 36), i.e., the viewpoint position, line of sight direction, and viewing angle used to generate the virtual viewpoint image 64, are specified by, for example, the viewpoint position determined based on coordinates extracted from the three-dimensional area image 32, the line of sight direction specified in advance by the user 13, etc., and the viewing angle specified in advance by the user 13, etc.

[0075] In addition, when the virtual viewpoint image 64 includes a pointing position identified from the pointing position identification information 62A of the reference image 62 with pointing position, the CPU 22A assigns a mark 61 (in the example shown in Figure 8, a dashed cross mark) that can identify the pointing position within the virtual viewpoint image 64 to the pointing position.

[0076] Mark 61 is an example of a "second mark" according to the technology of the present disclosure. The viewpoint position, line of sight direction, and field of view angle used to generate virtual viewpoint image 64 are an example of an "observation mode" according to the technology of the present disclosure. The area defined by the viewpoint position, line of sight direction, and field of view angle used to generate virtual viewpoint image 64 is an example of a "specified area within a three-dimensional area" according to the technology of the present disclosure. The viewpoint position used to generate virtual viewpoint image 64 corresponds to the position from which user 13 virtually observes the inside of soccer stadium 36, and is an example of an "observation position" according to the technology of the present disclosure.

[0077] Here, a moving image using 3D polygons generated based on a plurality of captured images obtained by capturing images of the inside of the soccer stadium 36 using a plurality of imaging devices 30 is applied as an example of the virtual viewpoint image 64. Note that the moving image is merely an example, and a still image may also be used.

[0078] The CPU 22A transmits the virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24 (see FIG. 1). The virtual viewpoint image 64 transmitted in this manner is received by the user device 12 and displayed on the display 18 as a new reference image 60 (see FIG. 8).

[0079] Next, the operation of the information processing system 2 will be described.

[0080] First, an example of the flow of user device-side processing performed by the CPU 40A of the user device 12 will be described with reference to FIG.

[0081] In the user device-side processing shown in FIG. 11, first, in step ST10, the CPU 40A acquires a live view image from the imaging device 42, and then the user device-side processing proceeds to step ST12.

[0082] In step ST12, the CPU 40A generates a reference image 60 based on the live view image acquired in step ST10, and then the user device side processing proceeds to step ST14.

[0083] In step ST14, the CPU 40A causes the reference image 60 generated in step ST12 to be displayed on the display 18, and thereafter the user device side processing proceeds to step ST16.

[0084] In step ST16, the CPU 40A determines whether the pointed position has been confirmed. Here, if a confirmation instruction is received by the touch panel 20, it is determined that the pointed position has been confirmed; if a confirmation instruction is not received by the touch panel 20, it is determined that the pointed position has not been confirmed. In step ST16, if the pointed position has not been confirmed, the determination is negative, and the user device-side processing proceeds to step ST28. In step ST16, if the pointed position has been confirmed, the determination is positive, and the user device-side processing proceeds to step ST18.

[0085] In step ST28, the CPU 40A determines whether a condition for terminating the user device-side process (hereinafter referred to as the "user device-side process termination condition") has been satisfied. A first example of the user device-side process termination condition is that an instruction to terminate the user device-side process has been accepted by the accepting device 50. A second example of the user device-side process termination condition is that a first predetermined time (e.g., 60 minutes) has elapsed since the execution of the user device-side process started. A third example of the user device-side process termination condition is that the processing capacity of the CPU 40A has fallen below a reference level.

[0086] If the user device-side process termination condition is not satisfied in step ST28, the determination is negative, and the user device-side process proceeds to step ST10. If the user device-side process termination condition is satisfied in step ST28, the determination is positive, and the user device-side process ends.

[0087] In step ST18, the CPU 40A generates a reference image 62 with a pointing position based on the reference image 60 generated in step ST12 or the reference image 60 with a target mark 60A added to a virtual viewpoint image 64 received by the transmission / reception device 44 in step ST20 (described later).The CPU 40A then transmits the generated reference image 62 with a pointing position to the information processing device 10 via the transmission / reception device 44. After the processing of step ST18 is executed, the user device-side processing proceeds to step ST20.

[0088] In step ST20, the CPU 40A determines whether or not the virtual viewpoint image 64 transmitted from the information processing device 10 has been received by the transmission / reception device 44 as a result of execution of the processing of step ST60 of the image generation processing shown in Fig. 12. If the virtual viewpoint image 64 has not been received by the transmission / reception device 44 in step ST20, the determination is negative, and the determination of step ST20 is made again. If the virtual viewpoint image 64 has been received by the transmission / reception device 44 in step ST20, the determination is positive, and the user device-side processing proceeds to step ST22.

[0089] In step ST22, the CPU 40A causes the display 18 to display the virtual viewpoint image 64 received by the transmitting / receiving device 44 in step ST20 as a new reference image 60, and then the user device side processing proceeds to step ST24.

[0090] In step ST24, the CPU 40A determines whether the designated position has been confirmed. If the designated position has not been confirmed in step ST24, the determination is negative, and the user device-side processing proceeds to step ST26. If the designated position has been confirmed in step ST24, the determination is positive, and the user device-side processing proceeds to step ST18.

[0091] In step ST26, the CPU 40A determines whether the user device-side processing termination condition is satisfied. If the user device-side processing termination condition is not satisfied in step ST26, the determination is negative, and the user device-side processing proceeds to step ST24. If the user device-side processing termination condition is satisfied in step ST26, the determination is positive, and the user device-side processing ends.

[0092] Next, an example of the flow of image generation processing performed by the CPU 22A of the information processing device 10 will be described with reference to Fig. 12. Note that the flow of image generation processing shown in Fig. 12 is an example of an "information processing method" according to the technique of the present disclosure.

[0093] In the image generation process shown in Fig. 12, first, in step ST50, the CPU 22A determines whether or not the reference image 62 with the designation position, which was transmitted by executing the process of step ST18 of the user device-side process shown in Fig. 11, has been received by the transmission / reception device 24. If the reference image 62 with the designation position has not been received by the transmission / reception device 24 in step ST50, the determination is negative, and the image generation process proceeds to step ST62. If the reference image 62 with the designation position has been received by the transmission / reception device 24 in step ST50, the determination is positive, and the image generation process proceeds to step ST52.

[0094] In step ST52, the CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and then the image generation process proceeds to step ST54.

[0095] In step ST54, the CPU 22A compares the reference image 62 with the designated position received by the transmitting / receiving device 24 in step ST50 with the three-dimensional area image 32 acquired in step ST52, and then the image generation process proceeds to step ST56.

[0096] In step ST56, the CPU 22A uses the comparison result between the reference image 62 with the pointing position and the three-dimensional area image in step ST54 to extract coordinates corresponding to the pointing position identified from the pointing position identification information 62A of the reference image 62 with the pointing position from the three-dimensional area image 32, and then the image generation process proceeds to step ST58.

[0097] In step ST58, the CPU 22A generates a virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted in step ST56, and then the image generation process proceeds to step ST60.

[0098] In step ST60, the CPU 22A transmits the virtual viewpoint image 64 generated in step ST58 to the user device 12 via the transmission / reception device 24, and then the image generation process proceeds to step ST62.

[0099] In step ST62, the CPU 22A determines whether a condition for terminating the image generation process (hereinafter referred to as the "image generation process termination condition") has been satisfied. A first example of the image generation process termination condition is that an instruction to terminate the image generation process has been given to the information processing device 10 by an administrator of the information processing device 10 or the like. A second example of the image generation process termination condition is that a second predetermined time (e.g., 10 hours) has elapsed since the execution of the image generation process started. A third example of the image generation process termination condition is that the processing capacity of the CPU 22A has decreased to below a reference level.

[0100] In step ST62, if the image generation process termination condition is not satisfied, the determination is negative and the image generation process proceeds to step ST50. If the image generation process termination condition is satisfied, the determination is positive and the image generation process ends.

[0101] As described above, in the information processing system 2, the CPU 22A acquires a virtual viewpoint image 64 that shows a subject present in the soccer stadium 36 when the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position that is determined based on coordinates within the soccer stadium 36 that correspond to a designated position designated in a reference image 60 (see FIG. 6) that shows the state of the inside of the soccer stadium 36 when the user 13 observes the soccer stadium 36 (see FIG. 4) from the viewpoint position 56 (see FIG. 4). The virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, this configuration allows the user 13 to observe the state of the inside of the soccer stadium 36 from various positions.

[0102] Furthermore, in the information processing system 2, the CPU 22A derives the coordinates within the soccer stadium 36 that correspond to the indicated position based on the observation manner in which the user 13 observes the inside of the soccer stadium 36 and the indicated position. Therefore, with this configuration, it is possible to identify which position within the soccer stadium 36 the position indicated by the user 13 is.

[0103] Furthermore, in the information processing system 2, the observation manner in which the user 13 observes the inside of the soccer stadium 36 is determined according to the position from which the user 13 observes the inside of the soccer stadium 36 in real space or virtual space. Therefore, when the position from which the user 13 observes the inside of the soccer stadium 36 changes, the observation manner in which the user 13 observes the inside of the soccer stadium 36 also changes accordingly. In this case, too, the CPU 22A derives coordinates within the soccer stadium 36 that correspond to the indicated position based on the observation manner in which the user 13 observes the inside of the soccer stadium 36 and the indicated position. Therefore, with this configuration, even if the observation manner changes as the position from which the user 13 observes the inside of the soccer stadium 36 changes, it is possible to identify which position within the soccer stadium 36 the position indicated by the user 13 is.

[0104] Furthermore, in the information processing system 2, the CPU 22A derives coordinates within the soccer stadium 36 that correspond to the indicated position based on the correspondence between the live view image or the virtual viewpoint image 64 and the three-dimensional area image 32. Therefore, with this configuration, it is possible to identify with high accuracy the position within the soccer stadium 36 that the position indicated by the user 13 is, compared to a case where the position within the soccer stadium 36 that the position indicated by the user 13 is estimated based only on information obtained visually from the live view image or the virtual viewpoint image 64 and human intuition.

[0105] Furthermore, in the information processing system 2, an image based on a live view image and an image based on a virtual viewpoint image 64 are used as the reference image 60. Therefore, according to this configuration, the user 13 can specify the viewpoint position while checking the state in the real space, or can specify the viewpoint position while checking the state in the virtual space.

[0106] Furthermore, in the information processing system 2, the position of the target mark 60A in the live view image or the virtual viewpoint image 64 is the position specified by the user 13. The position of the target mark 60A is changed in accordance with a change instruction (see FIG. 6) given by the user 13, and is confirmed in accordance with a confirmation instruction (see FIG. 7) given by the user 13. Therefore, with this configuration, the position intended by the user 13 can be set as the viewpoint position.

[0107] Furthermore, in the information processing system 2, a target mark 60A is used as a mark that can identify the pointed position in the reference image 60, and the target mark 60A is displayed on the display 18 of the user device 12 as being included in the reference image 60. Therefore, according to this configuration, the pointed position in the reference image 60 can be visually recognized by the user 13.

[0108] Furthermore, in the information processing system 2, the virtual viewpoint image 64 includes a mark 61. The mark 61 is a mark that allows the user 13 to identify the designated position used to generate the virtual viewpoint image 64 within the virtual viewpoint image 64. Therefore, according to this configuration, the user 13 can infer the designated position used to generate the virtual viewpoint image 64 from the virtual viewpoint image 64.

[0109] Furthermore, in the information processing system 2, the coordinates relating to the position of the soccer field 36A shown in the three-dimensional area image 32 are set to be coordinates indicating a position higher than the actual position of the soccer field 36A. Therefore, with this configuration, it is possible to prevent the viewpoint position from being set on the ground of the soccer field 36A.

[0110] Furthermore, in the information processing system 2, the CPU 22A detects the pointed position based on the reference image 60. That is, the position of the target mark 60A is detected as the pointed position by the CPU 22A. Therefore, with this configuration, the pointed position can be identified with higher accuracy than when the pointed position is estimated based only on information obtained visually from the live view image or the virtual viewpoint image 64 and human intuition.

[0111] In the first embodiment, an example has been described in which the CPU 22A generates the virtual viewpoint image 64 in the image generation process and transmits it to the user device 12. However, the technology of the present disclosure is not limited to this. For example, when the imaging device 30 is installed at a position that coincides with the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32, a captured image obtained by capturing an image using the imaging device 30 installed at the position that coincides with the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32 may be transmitted from the information processing device 10 to the user device 12 as a substitute image for the virtual viewpoint image 64.

[0112] Furthermore, in the first embodiment described above, an example has been given in which the CPU 22A generates a new virtual viewpoint image 64 and transmits it to the user device 12 each time it acquires a reference image 62 with a pointing position, but the technology of the present disclosure is not limited to this. For example, if an object image showing an object present in the soccer stadium 36 when the soccer stadium 36 is observed from a position whose distance from the pointing position identified from the pointing position identification information 62A of the reference image 62 with a pointing position is within a threshold range, is stored in the storage area, the CPU 22A may acquire the object image from the storage area and transmit the acquired object image to the user device 12.

[0113] In this case, as shown in FIG. 13 as an example, the CPU 22A extracts coordinates corresponding to the pointing position identified from the pointing position identification information 62A of the pointing-position-attached reference image 62 from the three-dimensional area image 32, as in the first embodiment. Then, the CPU 22A determines whether a virtual viewpoint image 64 associated with coordinates within a neighborhood of the coordinates extracted from the three-dimensional area image 32 is stored in the NVM 22B. Here, the neighborhood refers to, for example, a range within a radius of two meters. The radius of two meters is an example of a "threshold" according to the technology of the present disclosure. The NVM 22B is an example of a "storage area" according to the technology of the present disclosure. The virtual viewpoint image 64 is an example of an "object image" according to the technology of the present disclosure, and the object shown in the virtual viewpoint image 64 is an example of an "object" according to the technology of the present disclosure.

[0114] If the virtual viewpoint image 64 associated with the coordinates within the vicinity range of the coordinates extracted from the three-dimensional area image 32 by the CPU 22A is not stored in the NVM 22B, the CPU 22A generates the virtual viewpoint image 64 in the same manner as in the first embodiment, and transmits the generated virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24. The CPU 22A also associates the coordinates extracted from the three-dimensional area image 32 with the generated virtual viewpoint image 64, and stores the virtual viewpoint image 64 associated with the coordinates in the NVM 22B.

[0115] On the other hand, when the virtual viewpoint image 64 associated with coordinates within a range close to the coordinates extracted from the three-dimensional area image 32 by the CPU 22A is stored in the NVM 22B, as shown in FIG. 14 as an example, the CPU 22A acquires from the NVM 22B the virtual viewpoint image 64 associated with coordinates closest to the coordinates extracted from the three-dimensional area image 32. Then, the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24. This makes it possible to provide the virtual viewpoint image 64 to the user 13 more quickly than when the CPU 22A generates a new virtual viewpoint image 64 every time it acquires a reference image 62 with a designated position.

[0116] In the examples shown in Figures 13 and 14, an example is described in which a virtual viewpoint image 64 is generated and transmitted to the user device 12 or stored in NVM 22B, but the technology of the present disclosure is not limited to this, and a captured image may be transmitted to the user device 12 or stored in NVM 22B together with or instead of the virtual viewpoint image 64.

[0117] [Second embodiment] In the first embodiment, a case was described in which the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates within the soccer stadium 36 corresponding to a position indicated in the reference image 60, but in the second embodiment, a case will be described in which the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates within the soccer stadium 36 corresponding to a position indicated in the soccer stadium 36 that is the observation target. In the second embodiment, the same components as in the first embodiment are assigned the same reference numerals, and their description will be omitted, and only parts that differ from the first embodiment will be described.

[0118] As an example, as shown in FIG. 15, an information processing system 66 includes an information processing device 10, a user device 12, and an HMD 68.

[0119] The HMD 68 includes an HMD main body 70 and a band 72. The band 72 is a stretchable member formed in a strip shape from one end to the other end of the HMD main body 70. The HMD 68 has an annular outer shape formed by the HMD main body 70 and the band 72, and is fixed in a state of tight contact with the upper half of the head of the user 13.

[0120] The HMD main body 70 has a display 74, an HMD camera 76, a computer 78, a reception device 80, and a transmission / reception device 82. The display 74 has a screen (not shown) and a projection unit (not shown). The screen is made of a transparent material, and the user 13 views the real space with the naked eye through the screen. In other words, the HMD 68 is a see-through HMD. The HMD main body 70 does not necessarily have to include the computer 78; the computer 78 may be provided separately from the HMD main body 70. In this case, the HMD main body 70 may only have the function of displaying data received from the computer 78 via the transmission / reception device 82 and transmitting data related to images captured by the HMD camera 76 to the computer 78. The HMD camera 76 may also be provided separately from the HMD main body 70. For example, the HMD camera 76 may be a camera that is detachable from the HMD main body 70.

[0121] The screen is positioned directly opposite the eyes of the user 13, and an image is projected onto the inner surface of the screen (the surface facing the user 13) by a projection unit. The projection unit is a well-known device and will not be described in detail here; however, it is a device that includes a display element such as a liquid crystal display that displays an image, and a projection optical system that projects the image displayed on the display element onto the inner surface of the screen. The screen is realized by a half mirror that reflects the image projected by the projection unit and transmits light from the real space. The projection unit projects the image onto the inner surface of the screen at a predetermined frame rate (e.g., 60 frames per second). The image is reflected by the inner surface of the screen and enters the eyes of the user 13. This allows the user 13 to visually recognize the image.

[0122] The HMD camera 76 is an imaging device that has a CMOS image sensor and is equipped with an optical zoom function and / or a digital zoom function. Note that other types of image sensors, such as a CCD image sensor, may be used instead of a CMOS image sensor. The HMD camera 76 is positioned in front of the forehead of the user 13 and captures images in front of the user 13.

[0123] The computer 78 includes a CPU 78A, an NVM 78B, and a RAM 78C, which are connected to each other via a bus 84. In the example shown in Fig. 15, for convenience of illustration, a single bus is shown as the bus 84, but multiple buses may be used. The bus 84 may also include a serial bus or a parallel bus made up of a data bus, an address bus, a control bus, and the like.

[0124] The CPU 78A controls the entire HMD main body 70. The NVM 78B stores various parameters, various programs, etc. An example of the NVM 78B is an EEPROM. Various information is temporarily stored in the RAM 78C. The RAM 78C is used as a work memory by the CPU 78A.

[0125] The display 74 is connected to the bus 84. Specifically, the above-mentioned projection unit is connected to the bus 84. The display 74 displays various information under the control of the CPU 78A.

[0126] The HMD camera 76 is connected to a bus 84, and the CPU 78A controls the HMD camera 76. A captured image obtained by capturing an image with the HMD camera 76 is acquired via the bus 84 by the CPU 78A.

[0127] The transmitting / receiving device 82 is connected to the bus 84. The transmitting / receiving device 82 is a device including a communication processor (not shown), an antenna, etc., and transmits and receives various types of information to and from the information processing device 10 via a base station (not shown) under the control of the CPU 78A. That is, the CPU 78A transmits and receives various types of information to and from the information processing device 10 via the transmitting / receiving device 82.

[0128] The accepting device 80 is a device including at least one hard key, and accepts instructions from the user 13. The accepting device 80 is connected to the bus 84, and the CPU 78A acquires the instructions accepted by the accepting device 80.

[0129] The NVM 78B stores an HMD-side processing program 85. The CPU 78A reads the HMD-side processing program 85 from the NVM 78B and executes the HMD-side processing program 85 on the RAM 78C to perform HMD-side processing (see FIG. 28).

[0130] 16 as an example, in the HMD-side processing, the CPU 78A acquires an HMD image 86 (see FIG. 17) from the HMD camera 76. The HMD image 86 is, for example, a live view image. The CPU 78A causes the display 74 to display the HMD image 86.

[0131] 17 as an example, the user 13 can observe the real space through the entire display 74. An HMD image 86 is displayed on the display 74. The HMD image 86 is displayed by the user 13 through the display 74 in a state where it is superimposed on a part of the real space area. Since light from the real space also passes through the display area of ​​the HMD image 86, the user 13 can observe the real space through the display area of ​​the HMD image 86.

[0132] 18 , when the finger of the user 13 enters the field of view of the HMD camera 76, the user's finger is captured by the HMD camera 76 and displayed as an image in the HMD image 86. By pointing their finger within the field of view of the HMD camera 76, the user 13 tentatively designates the position pointed at with their finger as the viewpoint from which the user 13 observes the inside of the soccer stadium 36. Note that the actual direction in which the finger is pointed is offset from the line of sight of the user 13, and therefore the point at which the finger is pointed is not the position that the user 13 intends as the viewpoint. The position that the user 13 intends as the viewpoint is the end of the line of sight that passes through the tip of the user's 13's finger. The direction of the optical axis OA of the imaging optical system of the HMD camera 76 is approximately aligned with the line of sight of the user 13. Therefore, the information processing system 66 determines the position within the soccer stadium 36 that meets the optical axis OA when the center of the angle of view and the position of the user's 13's fingertip coincide, that is, the point (gazing point) that the user 13 is currently gazing at while observing the soccer stadium 36, as the position provisionally indicated by the user 13 as the viewpoint position from which the user 13 observes the soccer stadium 36 (hereinafter also referred to as the "provisionally indicated position").

[0133] As an example, as shown in FIG. 19, in HMD-side processing, the CPU 78A detects the finger of the user 13 using an HMD image 86 obtained by capturing an image with the HMD camera 76. The CPU 78A generates an HMD image 88 (see FIG. 20) with a tentative pointing position when the fingertip of the user 13 comes to rest at the center of the angle of view. The case where the fingertip of the user 13 comes to rest at the center of the angle of view means, for example, that the state in which the fingertip of the user 13 comes to rest at the center of the angle of view continues for a pre-specified time (e.g., 3 seconds). The CPU 78A transmits the generated HMD image 88 with a tentative pointing position to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).

[0134] 20, an HMD image 88 with a tentative pointing position is an image in which tentative pointing position identification information 88A is added to an HMD image 86. The tentative pointing position identification information 88A refers to information that can identify a tentative pointing position in the HMD image 86 (for example, information that can identify the position of a pixel corresponding to the tentative pointing position in the HMD image 86, i.e., information that can identify the position of a pixel corresponding to the position of a fingertip captured in the HMD image 86).

[0135] 21 as an example, in the image generation process of the information processing device 10, the CPU 22A acquires an HMD image 88 with a tentative pointing position from the HMD 68. The CPU 22A acquires, from the imaging device 30, a captured image showing a subject including a tentative pointing position identified from tentative pointing position identification information 88A of the HMD image 88 with the tentative pointing position, as a different viewpoint position image 90 (see FIG. 22) showing the state of the inside of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position different from the current viewpoint position of the user 13.

[0136] The CPU 22A acquires the three-dimensional area image 32 from the NVM 22B and generates a different viewpoint position image 92 with designation position candidates (see FIG. 22 ) by referring to the acquired three-dimensional area image 32. Specifically, the CPU 22A first identifies matching feature points between the HMD image 88 with the temporary designation position and the three-dimensional area image 32, thereby identifying the position of the optical axis OA within the soccer stadium 36. Next, the CPU 22A compares the HMD image 88 with the temporary designation position with the three-dimensional area image 32, and extracts coordinates of the temporary designation position within the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. Next, the CPU 22A generates a plurality of designation position candidates. The plurality of designation position candidates are positions defined on the optical axis OA at predetermined intervals (for example, at intervals of 5 meters on a real space scale), including the temporary designation position. Coordinates obtained from the three-dimensional area image 32 are associated with each of the plurality of designation position candidates. Then, the CPU 22A generates an other viewpoint position image 92 with pointing position candidates (see FIG. 22) by adding information such as a plurality of pointing position candidates to the other viewpoint position image 90. The CPU 22A transmits the other viewpoint position image 92 with pointing position candidates to the HMD 68 via the transmission / reception device 24 (see FIG. 1).

[0137] The optical axis OA is an example of a "first line" according to the technology of the present disclosure, and the tentatively indicated position is an example of a "point of gaze" according to the technology of the present disclosure.

[0138] 22, the other viewpoint position image 92 with pointing position candidate is an image in which a plurality of dot marks 92A and a message 92B are superimposed on the other viewpoint position image 90. The plurality of dot marks 92A are arranged at predetermined intervals on the image indicating the optical axis OA, and each dot mark 92A is a mark that can identify a pointing position candidate. Note that it is not essential to display the image indicating the optical axis OA, and the image indicating the optical axis OA does not have to be displayed.

[0139] Each dot mark 92A is associated with coordinates that can identify the position of the designated position candidate, which are obtained from the three-dimensional area image 32. The message 92B is a message that prompts the user 13 to select a designated position candidate, and in the example shown in Fig. 22, an example of the message 92B is a message that reads, "Please specify any dot (position)."

[0140] 23 as an example, in the HMD-side processing, the CPU 78A acquires an other viewpoint position image 92 with pointing position candidates from the information processing device 10. Then, as shown in FIG. 24 as an example, the CPU 78A causes the display 74 to display the other viewpoint position image 92 with pointing position candidates.

[0141] The user 13 indicates the intended observation position within the soccer stadium 36 by positioning his / her fingertip on one of the multiple dot marks 92A included in the other-viewpoint position image 92 with the indication position candidate. That is, the user 13 indicates the multiple indication position candidates provided on the optical axis OA, thereby determining the intended observation position of the user 13. Note that the observation position intended by the user 13 here is an example of the "indicated position," "observation position," and "indicated position on the first line" according to the technology of the present disclosure.

[0142] 25 as an example, in the HMD-side processing, the CPU 78A acquires an HMD image 86 from the HMD camera 76 and detects the finger of the user 13 using the acquired HMD image 86. When the fingertip of the user 13 is positioned on any of the dot marks 92A, the CPU 78A transmits information including coordinates associated with the dot mark 92A on which the fingertip of the user 13 is positioned as pointing position identification information 94 to the information processing device 10 via the transmission / reception device 82 (see FIG. 15). The pointing position identification information 94 is information that can identify the pointing position candidate selected by the user 13 via the dot mark 92A, that is, information that can identify the position indicated by the user 13 as the viewpoint position from which the user 13 observes the inside of the soccer stadium 36.

[0143] As an example, as shown in FIG. 26 , in the image generation process of the information processing device 10, the CPU 22A acquires pointing position identification information 94 from the HMD 68. The CPU 22A extracts coordinates from the pointing position identification information 94 and generates a virtual viewpoint image 64 (see FIG. 10 ) using a viewpoint position determined based on the extracted coordinates. The CPU 22A transmits the generated virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24. As a result, the virtual viewpoint image 64 is displayed on the display 18 of the user device 12, similar to the first embodiment.

[0144] Note that, although an example is given here in which the virtual viewpoint image 64 is transmitted to the user device 12, the technology of the present disclosure is not limited to this, and the virtual viewpoint image 64 may be transmitted to the HMD 68 and displayed on the display 74 of the HMD 68.

[0145] Next, the operation of the information processing system 66 will be described.

[0146] First, an example of the flow of user device-side processing performed by the CPU 40A of the user device 12 will be described with reference to FIG.

[0147] 27, first, in step ST100, the CPU 40A determines whether or not the virtual viewpoint image 64 transmitted by executing the processing of step ST214 of the image generation processing shown in Fig. 30 has been received by the transmission / reception device 44. If the virtual viewpoint image 64 has not been received by the transmission / reception device 44 in step ST100, the determination is negative, and the user device-side processing proceeds to step ST104. If the virtual viewpoint image 64 has been received by the transmission / reception device 44 in step ST100, the determination is positive, and the user device-side processing proceeds to step ST102.

[0148] In step ST102, the CPU 40A causes the display 18 to display the virtual viewpoint image 64 received by the transmitting / receiving device 44 in step ST100, and then the user device side processing proceeds to step ST104.

[0149] In step ST104, the CPU 40A determines whether the user device-side processing termination condition is satisfied. If the user device-side processing termination condition is not satisfied in step ST104, the determination is negative, and the user device-side processing proceeds to step ST100. If the user device-side processing termination condition is satisfied in step ST104, the determination is positive, and the user device-side processing ends.

[0150] Next, an example of the flow of HMD-side processing performed by the CPU 78A of the HMD 68 will be described with reference to FIGS.

[0151] In the HMD-side processing shown in FIG. 28, first, in step ST150, the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD-side processing proceeds to step ST152.

[0152] In step ST152, the CPU 78A causes the display 74 to display the HMD image 86 acquired in step ST150, and then the HMD-side processing proceeds to step ST154.

[0153] In step ST154, the CPU 78A executes finger detection processing using the HMD image 86 acquired in step ST152. The finger detection processing refers to processing for detecting the finger of the user 13 using the HMD image 86. After the processing of step ST154 is executed, the HMD-side processing proceeds to step ST156.

[0154] In step ST156, the CPU 78A determines whether or not a finger of the user 13 is detected by the finger detection processing of step ST154. If a finger of the user 13 is not detected by the finger detection processing of step ST154 in step ST156, the determination is negative, and the process proceeds to step ST178 shown in Fig. 29. If a finger of the user 13 is detected by the finger detection processing of step ST154 in step ST156, the determination is positive, and the HMD-side process proceeds to step ST158.

[0155] In step ST158, the CPU 78A determines whether or not the fingertip of the user 13 is resting at the center of the angle of view of the HMD camera 76. If the fingertip of the user 13 is not resting at the center of the angle of view of the HMD camera 76 in step ST158, the determination is negative, and the HMD-side processing proceeds to step ST150. If the fingertip of the user 13 is resting at the center of the angle of view of the HMD camera 76 in step ST158, the determination is positive, and the HMD-side processing proceeds to step ST160.

[0156] In step ST160, the CPU 78A generates an HMD image 88 with a tentative designated position based on the HMD image 86 acquired in step ST150, and then the HMD-side processing proceeds to step ST162.

[0157] In step ST162, the CPU 78A transmits the HMD image 88 with the tentative designated position generated in step ST160 to the information processing device 10 via the transmission / reception device 82, and then the HMD-side processing proceeds to step ST164.

[0158] In step ST164, the CPU 78A determines whether or not the other viewpoint position image 92 with the pointing position candidate transmitted by executing the processing of step ST206 shown in Fig. 30 has been received by the transmission / reception device 82. If the other viewpoint position image 92 with the pointing position candidate has not been received by the transmission / reception device 82 in step ST164, the determination is negative, and the determination of step ST164 is made again. If the other viewpoint position image 92 with the pointing position candidate has been received by the transmission / reception device 82 in step ST164, the determination is positive, and the HMD-side processing proceeds to step ST166.

[0159] In step ST166, the CPU 78A causes the display 74 to display the other viewpoint position image 92 with the indication position candidate received by the transmitting / receiving device 82 in step ST164, and then the HMD-side processing proceeds to step ST168 shown in FIG.

[0160] In step ST168 shown in FIG. 29, the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD-side processing proceeds to step ST170.

[0161] In step ST170, the CPU 78A executes finger detection processing using the HMD image 86 acquired in step ST168, and then the HMD-side processing proceeds to step ST172.

[0162] In step ST172, the CPU 78A determines whether or not a finger of the user 13 is detected by the finger detection processing of step ST170. If a finger of the user 13 is not detected by the finger detection processing of step ST170 in step ST172, the determination is negative, and the process proceeds to step ST180. If a finger of the user 13 is detected by the finger detection processing of step ST170 in step ST172, the determination is positive, and the HMD-side process proceeds to step ST174.

[0163] In step ST180, the CPU 78A determines whether or not a condition for terminating the HMD-side processing (hereinafter referred to as the "HMD-side processing termination condition") has been satisfied. A first example of the HMD-side processing termination condition is that an instruction to terminate the HMD-side processing has been accepted by the accepting device 80. A second example of the HMD-side processing termination condition is that a third predetermined time (e.g., 60 minutes) has elapsed since the execution of the HMD-side processing started. A third example of the HMD-side processing termination condition is that the processing capacity of the CPU 78A has fallen below a reference level.

[0164] In step ST180, if the HMD-side processing end condition is not satisfied, the determination is negative, and the HMD-side processing proceeds to step ST168. In step ST180, if the HMD-side processing end condition is satisfied, the determination is positive, and the HMD-side processing ends.

[0165] In step ST174, the CPU 78A determines whether or not the fingertip (the fingertip of the user 13) detected in step ST172 is located at the dot mark 92A on the other viewpoint position image 92 with pointing position candidates displayed on the display 74. If the fingertip of the user 13 is not located at the dot mark 92A on the other viewpoint position image 92 with pointing position candidates in step ST174, the determination is negative, and the HMD-side processing proceeds to step ST168. If the fingertip of the user 13 is located at the dot mark 92A on the other viewpoint position image 92 with pointing position candidates in step ST174, the determination is positive, and the HMD-side processing proceeds to step ST176.

[0166] In step ST176, the CPU 78A transmits information including the coordinates associated with the dot mark 92A where the fingertip of the user 13 is located as instruction position identification information 94 to the information processing device 10 via the transmission / reception device 82, and then the HMD-side processing proceeds to step ST178.

[0167] In step ST178, the CPU 78A determines whether or not the HMD-side processing end condition is satisfied. If the HMD-side processing end condition is not satisfied in step ST178, the determination is negative, and the HMD-side processing proceeds to step ST150 shown in Fig. 28. If the HMD-side processing end condition is satisfied in step ST178, the determination is positive, and the HMD-side processing ends.

[0168] Next, an example of the flow of image generation processing performed by the CPU 22A of the information processing device 10 will be described with reference to FIG.

[0169] In the image generation process shown in Fig. 30, first, in step ST200, the CPU 22A determines whether or not the HMD image 88 with the temporary pointing position, which was transmitted by executing the process of step ST162 of the HMD-side process shown in Fig. 28, has been received by the transmission / reception device 24. If the HMD image 88 with the temporary pointing position has not been received by the transmission / reception device 24 in step ST200, the determination is negative, and the image generation process proceeds to step ST208. If the HMD image 88 with the temporary pointing position has been received by the transmission / reception device 24 in step ST200, the determination is positive, and the image generation process proceeds to step ST202.

[0170] In step ST202, the CPU 22A acquires from the imaging device 30 a captured image showing a subject including the tentative pointing position identified from the tentative pointing position identification information 88A of the HMD image 88 with tentative pointing position received by the transmitting / receiving device 24 in step ST200, as a different viewpoint position image 90 showing the state of the inside of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position different from the current viewpoint position of the user 13. After the processing of step ST202 is executed, the image generation processing proceeds to step ST204.

[0171] In step ST204, the CPU 22A acquires the three-dimensional area image 32 from the NVM 22B, and generates a different viewpoint position image 92 with indication position candidates by referring to the acquired three-dimensional area image 32, after which the image generation process proceeds to step ST206.

[0172] In step ST206, the CPU 22A transmits the other viewpoint position image 92 with the indication position candidate generated in step ST202 to the HMD 68 via the transmission / reception device 24, and then the image generation process proceeds to step ST208.

[0173] In step ST208, the CPU 22A determines whether or not the designated position specifying information 94 transmitted by executing the processing of step ST176 shown in Fig. 29 has been received by the transmission / reception device 24. If the designated position specifying information 94 has not been received by the transmission / reception device 24 in step ST208, the determination is negative, and the image generation processing proceeds to step ST216. If the designated position specifying information 94 has been received by the transmission / reception device 24 in step ST208, the determination is positive, and the image generation processing proceeds to step ST210.

[0174] In step ST210, the CPU 22A extracts coordinates from the designated position specifying information 94 received by the transmitting / receiving device 24 in step ST208, and then the image generation process proceeds to step ST212.

[0175] In step ST212, the CPU 22A generates the virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted from the designated position specifying information 94 in step ST210, and then the image generation process proceeds to step ST214.

[0176] In step ST214, the CPU 22A transmits the virtual viewpoint image 64 generated in step ST212 to the user device 12 via the transmission / reception device 24, and then the image generation process proceeds to step ST216.

[0177] In step ST216, the CPU 22A determines whether or not the image generation process termination condition is satisfied. If the image generation process termination condition is not satisfied in step ST216, the determination is negative, and the image generation process proceeds to step ST200. If the image generation process termination condition is satisfied, the determination is positive, and the image generation process ends.

[0178] As described above, in the information processing system 66, the CPU 22A acquires a virtual viewpoint image 64 that shows a subject present in the soccer stadium 36 when the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on coordinates within the soccer stadium 36 that correspond to a position specified within the soccer stadium 36. The virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, this configuration allows the user 13 to observe the state of the inside of the soccer stadium 36 from various positions.

[0179] Furthermore, in the information processing system 65, the CPU 22A acquires a virtual viewpoint image 64 showing a subject present in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a designated position (e.g., any one of a plurality of candidate designated positions) on a line (e.g., on the optical axis OA) from a viewpoint at which the inside of the soccer stadium 36 is observed by the user 13 (e.g., the position of the HMD camera 76 in the soccer stadium 36) toward a gaze point (e.g., a tentative designated position) at which the user 13 is gazing, and the virtual viewpoint image 64 is displayed on the display 18 of the user device 12. Therefore, with this configuration, it is possible to generate a virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select a designated position corresponding to the viewpoint position used to generate the virtual viewpoint image 64 from a plurality of candidates.

[0180] In the second embodiment, the other viewpoint position image 92 with indication position candidates displayed on the display 74 is a two-dimensional image, but the technology of the present disclosure is not limited to this, and the image may be a stereoscopic image in which a plurality of dot marks 92A or marks that are alternatives to these (for example, star marks, triangular marks, or square marks) are visually recognizable via the display 74. The stereoscopic image may be generated based on a parallax image obtained using a plurality of phase difference pixels, or may be generated based on a vibration image obtained by applying vibration to the HMD camera 76.

[0181] Although the second embodiment has been described with reference to an example in which the CPU 22A of the information processing device 10 acquires the HMD image 88 with a tentative pointing position from the HMD 68 during the image generation process, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 31 , during the image generation process, the CPU 22A of the information processing device 10 may acquire a reference image 96 with a tentative pointing position from the user device 12. The reference image 96 with a tentative pointing position corresponds to the reference image 62 with a tentative pointing position described in the first embodiment and includes tentative pointing position identification information 96A. The tentative pointing position identification information 96A corresponds to the pointing position identification information 62A described in the first embodiment. In other words, the reference image 96 with a tentative pointing position is obtained by adding the pointing position identification information 62A described in the first embodiment to the reference image 60 as the tentative pointing position identification information 96A.

[0182] In this case, the CPU 22A acquires from the imaging device 30 an image showing a subject including a tentative instruction position (a position corresponding to the instruction position described in the first embodiment above) identified from the tentative instruction position identification information 96A of the reference image 96 with tentative instruction position, as a different viewpoint position image 90 showing the state of the inside of the soccer stadium 36 when observing the inside of the soccer stadium 36 from a viewpoint position different from the current viewpoint position of the user 13.

[0183] The CPU 22A acquires the three-dimensional area image 32 from the NVM 22B and generates an alternative viewpoint position image 92 with designation position candidates by referring to the acquired three-dimensional area image 32. Specifically, the CPU 22A first identifies matching feature points between the reference image 96 with temporary designation positions and the three-dimensional area image 32, thereby identifying the position of the optical axis OA within the soccer stadium 36. Next, the CPU 22A compares the reference image 96 with temporary designation positions with the three-dimensional area image 32 and extracts coordinates of a temporary designation position within the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. Next, the CPU 22A generates multiple designation position candidates using the method described in the second embodiment. The CPU 22A then generates an alternative viewpoint position image 92 with designation position candidates by adding information such as the multiple designation position candidates to the alternative viewpoint position image 90. The CPU 22A transmits the alternative viewpoint position image 92 with designation position candidates to the user device 12 via the transmission / reception device 24. In this case, an alternative viewpoint position image 92 with pointing position candidates is displayed on the display 18 of the user device 12. When one of the dot marks 92A is selected by the user 13 via the touch panel 20, the CPU 78A determines the position intended by the user 13 as the viewpoint position, and transmits pointing position identification information 94 to the information processing device 10, as in the second embodiment.

[0184] Furthermore, in the second embodiment described above, an example was given in which the fingers of the user 13 are detected based on the HMD image 86 obtained by capturing an image of the fingers of the user 13 using the HMD camera 76. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 32 , the fingers of the user 13 may be detected from multiple captured images captured by multiple imaging devices 30 installed in a soccer stadium 36. Furthermore, the fingers of the user 13 may be detected by the CPU 22A, CPU 78A, or the like based on the HMD image 86 and the multiple captured images. Furthermore, the method of detecting the fingers of the user 13 is not limited to the above. The fingers of the user 13 may be detected by attaching a known device whose position and direction can be identified to the user's finger. In this case, the user 13 may point to the viewpoint position using the finger to which the device is attached, thereby determining the viewpoint position in the same manner as in the above embodiment. Furthermore, the viewpoint position does not necessarily have to be determined by detecting the fingers of the user 13. For example, the viewpoint position may be determined in the same manner as in the above embodiment by having the user 13 hold a known device whose position and direction can be identified and point in a specific direction.

[0185] In the second embodiment, the viewpoint used to generate the virtual viewpoint image 64 is a designated position on a line extending from the viewpoint at which the soccer stadium 36 is observed to the point of interest, i.e., a designated position on the optical axis OA. However, the technology of the present disclosure is not limited to this. For example, the viewpoint used to generate the virtual viewpoint image 64 may be a designated position on a line extending from the viewpoint 56 (see FIG. 4) to a designated point in the image corresponding to the reference image 60 (e.g., in the line of sight of the user 13).

[0186] In this case, for example, as shown in Fig. 33, an alternative viewpoint position image 98 with pointing position candidates may be displayed on the display 18 of the user device 12. The alternative viewpoint position image 98 with pointing position candidates is an image equivalent to the alternative viewpoint position image 92 with pointing position candidates, and differs from the alternative viewpoint position image 92 with pointing position candidates in that a plurality of dot marks 92A are positioned not on an image indicating the optical axis OA but on an image indicating the line of sight 58A of the user 13. Displaying the image indicating the line of sight 58A is not essential, and the image indicating the line of sight 58A does not necessarily have to be displayed.

[0187] 4, the line of sight 58A corresponds to the optical axis of the imaging optical system of the imaging device 42. In this case, when the dot mark 92A is touched with the finger of the user 13 via the touch panel 20, a position identified from the coordinates associated with the touched dot mark 92A is set as the viewpoint position used for generating the virtual viewpoint image 64. Therefore, according to this configuration, it is possible to generate the virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select from a plurality of candidates an indication position that corresponds to the viewpoint position used for generating the virtual viewpoint image 64.

[0188] Furthermore, in the second embodiment, an example was described in which the CPU 22A associates the dot marks 92A with the pointing position candidates, but the technology of the present disclosure is not limited to this. For example, the CPU 22A may associate a thumbnail image 100B (see FIG. 36), which is a reduced version of the virtual viewpoint image 64 obtained when observing the inside of the soccer stadium 36, with the pointing position candidates. Here, how to associate the thumbnail image 100B with the pointing position candidates and use the thumbnail image 100B will be described with reference to FIGS. 34 to 38.

[0189] 34 as an example, in the image generation process of the information processing device 10, the CPU 22A generates a different viewpoint position image 100 with designated position candidates (see FIG. 36) by the method described in the second embodiment. The CPU 22A generates a plurality of virtual viewpoint images 64 using a plurality of viewpoint positions determined based on a plurality of coordinates associated with a plurality of designated position candidates included in the different viewpoint position image 100 with designated position candidates. The CPU 22A then stores each of the generated virtual viewpoint images 64 in the NVM 22B in a state where it is associated with the associated designated position candidate.

[0190] As an example, as shown in FIG. 35, the CPU 22A acquires a plurality of virtual viewpoint images 64 from the NVM 22B, and generates a plurality of thumbnail images 100B (see FIG. 36) corresponding to the plurality of virtual viewpoint images 64. The CPU 22A associates the thumbnail images 100B with each of a plurality of designation position candidates included in the other viewpoint position image 100 with designation position candidates. Then, the CPU 22A associates the thumbnail images 100B with the other viewpoint position image with designation position candidates. The other viewpoint position image 100 is transmitted to the HMD 68 via the transmitting / receiving device 24 (see FIG. 1). As a result, the other viewpoint position image 100 with the pointing position candidate is displayed on the display 74 of the HMD 68.

[0191] 36, as an example, a plurality of dot marks 92A are arranged at predetermined intervals along the image indicating the optical axis OA in another viewpoint position image 100 with indication position candidates. Displaying the image indicating the optical axis OA is not essential, and the image indicating the optical axis OA does not have to be displayed.

[0192] Furthermore, in the other viewpoint position image 100 with the pointing position candidate, thumbnail images 100B linked to each of the plurality of dot marks 92A are arranged along the image showing the optical axis OA. Furthermore, a message 100C is added to the other viewpoint position image 100 with the pointing position candidate. The message 100C is a message that prompts the user 13 to select a pointing position candidate, and in the example shown in Fig. 36, an example of the message 100C is a message that reads, "Please select one of the thumbnail images."

[0193] As an example, as shown in FIG. 37, when a different viewpoint position image 100 with candidate instruction positions is displayed on the display 74 of the HMD 68 and the fingertip of the user 13 is positioned on one of the thumbnail images 100B, the thumbnail image 100B on which the fingertip is positioned is transmitted by the CPU 78A to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).

[0194] 38 as an example, in the image generation processing of the information processing device 10, the CPU 22A acquires a thumbnail image 100B from the HMD 68, and acquires a virtual viewpoint image 64 corresponding to the acquired thumbnail image 100B from the NVM 22B. Then, the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24 (see FIG. 1). As a result, the virtual viewpoint image 64 is displayed on the display 18 of the user device 12.

[0195] 34 to 38, a thumbnail image 100B is associated with each of the multiple pointing position candidates included in the other viewpoint position image 100 with pointing position candidates, and the virtual viewpoint image 64 corresponding to the thumbnail image 100B selected by the user 13 is displayed on the display 18 of the user device 12. Therefore, with this configuration, it is possible to generate a virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select from multiple candidates an indication position corresponding to the viewpoint position used to generate the virtual viewpoint image 64. Furthermore, the user 13 can predict, via the thumbnail image 100B, what kind of virtual viewpoint image 64 will be provided by the information processing device 10.

[0196] 34 to 38, the user 13 selects one of the thumbnail images 100B while the other viewpoint position image 100 with the pointing position candidate is displayed on the display 74 of the HMD 68, but the technology of the present disclosure is not limited to this. For example, the user 13 may select a thumbnail image 100B associated with a pointing position candidate on a line (e.g., on an image showing the line of sight of the user 13) from the viewpoint position 56 (see FIG. 4) to a specified point (e.g., a tentative pointing position) in an image corresponding to the reference image 60.

[0197] In this case, for example, as shown in FIG. 39 , an alternative viewpoint position image 102 with pointing position candidates may be displayed on the display 18 of the user device 12. The alternative viewpoint position image 102 with pointing position candidates is an image equivalent to the alternative viewpoint position image 100 with pointing position candidates, and differs from the alternative viewpoint position image 100 in that multiple thumbnail images 100B are positioned on an image representing the line of sight 58A of the user 13, rather than on an image representing the optical axis OA. The line of sight 58A corresponds to the optical axis of the imaging optical system of the imaging device 42 in the example shown in FIG. 4 , for example. The thumbnail image 100B displayed on the display 18 of the user device 12 is an example of a “second reduced image” according to the technology of the present disclosure.

[0198] When the user 13 touches one of the multiple thumbnail images 100B with his / her finger via the touch panel 20 while the other viewpoint position image 102 with the pointing position candidate is displayed on the display 18 of the user device 12, the virtual viewpoint image 64 corresponding to the touched thumbnail image 100B is transmitted to the user device 12 and displayed on the display 18 of the user device 12. Therefore, according to this configuration, it is possible to generate a virtual viewpoint image 64 based on a viewpoint position that reflects the intention of the user 13 with high accuracy, compared to a case where there is no room to select from multiple candidates an indication position corresponding to the viewpoint position used to generate the virtual viewpoint image 64. Furthermore, the user 13 can predict, via the thumbnail image 100B, what kind of virtual viewpoint image 64 will be provided by the information processing device 10.

[0199] [Third embodiment] In the above embodiments, cases were described in which the indication positions could be set without restriction within the soccer stadium 36, but in this third embodiment, a case will be described in which the area in which the indication positions can be set is limited. In this third embodiment, the same components as those described in the above embodiments are given the same reference numerals, and their description will be omitted, and only differences from the above embodiments will be described.

[0200] As an example, as shown in Fig. 40, an information processing system 105 according to the third embodiment includes an information processing device 10 and a user device 12. In the information processing device 10, an observation range restriction processing program 104 and a three-dimensional area image 106 with spectator seat information are stored in the NVM 22B. The CPU 22A reads the observation range restriction processing program 104 from the NVM 22B and executes the observation range restriction processing program 104 on the RAM 22C to perform observation range restriction processing (see Fig. 47).

[0201] The three-dimensional area image 106 with spectator seat information is an image in which spectator seat information 106A (see FIG. 41) is added to the three-dimensional area image 32 described in the first embodiment. A plurality of three-dimensional area images 106 with spectator seat information are stored in the NVM 22B. The plurality of three-dimensional area images 106 with spectator seat information are images defined for each spectator venue, and are used separately for each spectator venue.

[0202] Spectator seats 36B (see FIG. 2) are divided into classes. The classes are determined according to the purchase price of the spectator ticket, and in the example shown in FIG. 41, the areas of spectator seats 36B are differentiated into classes S seats, A seats, and B seats. As an example, as shown in FIG. 41, the three-dimensional area image 106 with spectator seat information reflects the class of spectator seats 36B as spectator seat information 106A.

[0203] Spectator seat information 106A is information that includes class identification information that can identify the class of spectator seats 36B and coordinates that can identify each area divided into classes within the soccer stadium 36. A class is also assigned to the position of the spectator seats 36B where the user 13 is watching the game, and the user 13 can only watch the game in an area of ​​the same class. In other words, the information processing system 105 allows the user 13 to set a viewpoint position for an area within the soccer stadium 36 from which the user 13 can watch the game, but prohibits the user 13 from setting a viewpoint position for other areas.

[0204] Note that the user 13 is an example of an "instructor" according to the technology of the present disclosure, and the class of the spectator seats 36B is an example of an "attribute" according to the technology of the present disclosure.

[0205] In the information processing system 105, the grade assigned to the position of the spectator seats 36B where the user 13 is watching the game is identified based on a live view image captured by the user device 12.

[0206] 42, in the user device-side processing of the user device 12, the CPU 40A acquires a live view image from the imaging device 42. Then, the CPU 40A transmits the acquired live view image to the information processing device 10 via the transmission / reception device 44 (see FIG. 3).

[0207] As an example, as shown in FIG. 43 , in the observation range specification process of the information processing device 10, the CPU 22A acquires a live view image from the user device 12. The CPU 22A acquires a three-dimensional area image 106 with spectator seat information by referring to the live view image acquired from the user device 12. Specifically, the CPU 22A calculates the degree of match of feature points between the live view image and a three-dimensional area image 32 included in the three-dimensional area image 106 with spectator seat information, and selects and acquires one three-dimensional area image 106 with spectator seat information from the multiple three-dimensional area images 106 with spectator seat information based on the calculated degree of match. That is, the CPU 22A acquires the three-dimensional area image 106 with spectator seat information that includes the three-dimensional area image 32 with the greatest degree of match of feature points with the live view image. Here, the CPU 22A adds same-class area information to the three-dimensional area image 106 with spectator seat information. The same-rank area information is information (for example, coordinates) that can identify the same-rank area 110 (see FIG. 45) that is an area of ​​the same rank as the area where the user 13 is watching.

[0208] The same-class area information is generated by the CPU 22A based on the live view image and the three-dimensional area image 106 with spectator seat information. The CPU 22A identifies, from among the three-dimensional area images 32 included in the acquired three-dimensional area image 106 with spectator seat information, the image that most closely matches the live view image as a user viewing area image indicating the area where the user 13 is watching. The CPU 22A identifies the class corresponding to the user viewing area image by referring to the three-dimensional area image 106 with spectator seat information. The CPU 22A identifies an area of ​​the same class as the identified class, i.e., the same-class area 110, by referring to the three-dimensional area image 106 with spectator seat information. The CPU 22A adds same-class area information, which is information that can identify the identified same-class area 110, to the three-dimensional area image 106 with spectator seat information. Then, the CPU 22A transmits the three-dimensional area image 106 with spectator seat information to which the same-class area information has been added to the user device 12 via the transmission / reception device 24 (see FIG. 40).

[0209] As an example, as shown in FIG. 44 , in user device-side processing of the user device 12, the CPU 40A acquires a three-dimensional area image 106 with spectator seat information to which same-class area information has been added from the information processing device 10, and acquires a live-view image from the imaging device 42. The CPU 40A generates a reference image 108 (see FIG. 45 ) that is an image based on the three-dimensional area image 106 with spectator seat information to which same-class area information has been added. Specifically, the CPU 40A generates the reference image 108 using the live-view image by referring to the same-class area information and the three-dimensional area image 106 with spectator seat information. The CPU 40A causes the display 18 to display the reference image 108.

[0210] As an example, as shown in FIG. 45, the reference image 108 includes a target mark 60A and a same-class area 110. The same-class area 110 is shown in a manner that allows it to be distinguished from other areas in the reference image 108. In the example shown in FIG. 45, the same-class area 110 is outlined with a thick line. In the information processing system 105, the user 13 can set a viewpoint position only in the same-class area 110. Therefore, even if the user device 12 requests the information processing device 10 to set a viewpoint position in an area other than the same-class area 110 in the same manner as in the first embodiment, the information processing device 10 does not respond to the request to set a viewpoint position from the user device 12.

[0211] The three-dimensional area image 106 with spectator seat information to which the same-class area information is added is an example of a "three-dimensional area image" according to the technology of the present disclosure. The same-class area 110 is an example of an "observation position indication range" according to the technology of the present disclosure.

[0212] Next, the operation of the information processing system 105 will be described.

[0213] First, an example of the flow of user device-side processing performed by the CPU 40A of the user device 12 will be described with reference to Fig. 46. The flowchart shown in Fig. 46 differs from the flowchart shown in Fig. 11 in that steps ST250 to ST262 are added before step ST10 in the flowchart shown in Fig. 11. Only steps that differ from the flowchart shown in Fig. 11 will be described below.

[0214] In the user device-side processing shown in FIG. 46, first, in step ST250, the CPU 40A acquires a live view image from the imaging device 42, and then the user device-side processing proceeds to step ST252.

[0215] In step ST252, the CPU 40A transmits the live view image acquired in step ST250 to the information processing device 10 via the transmission / reception device 44 (see FIG. 3), and then the user device side processing proceeds to step ST254.

[0216] In step ST254, the CPU 40A determines whether the three-dimensional area image 106 with the same class area information and spectator seat information, which was transmitted by executing the processing of step ST304 shown in Fig. 47, has been received by the transmission / reception device 44 (see Fig. 3). If the three-dimensional area image 106 with spectator seat information has not been received by the transmission / reception device 44 in step ST254, the determination is negative, and the user device-side processing proceeds to step ST262. If the three-dimensional area image 106 with spectator seat information has been received by the transmission / reception device 44 in step ST254, the determination is positive, and the user device-side processing proceeds to step ST256.

[0217] In step ST262, the CPU 40A determines whether the user device-side processing termination condition is satisfied. If the user device-side processing termination condition is not satisfied in step ST262, the determination is negative, and the user device-side processing proceeds to step ST254. If the user device-side processing termination condition is satisfied in step ST262, the determination is positive, and the user device-side processing proceeds to step ST256.

[0218] In step ST256, the CPU 40A generates a reference image 108 using the live view image acquired in step ST256, by referring to the three-dimensional area image 106 with the same class area information and spectator seat information received by the transmission / reception device 44 in step ST254, and then the user device side processing proceeds to step ST260.

[0219] In step ST260, CPU 40A causes reference image 108 generated in step ST258 to be displayed on display 18, and thereafter, the user device side processing proceeds to step ST10 (see FIG. 11).

[0220] Next, an example of the flow of observation range restriction processing performed by the CPU 22A of the information processing device 10 will be described with reference to FIG.

[0221] In the observation range restriction process shown in Fig. 47, first, in step ST300, the CPU 22A determines whether or not the live view image transmitted by executing the process of step ST252 shown in Fig. 46 has been received by the transmission / reception device 24 (see Fig. 40). If the live view image has not been received by the transmission / reception device 24 in step ST300, the determination is negative, and the observation range restriction process proceeds to step ST306. If the live view image has been received by the transmission / reception device 24 in step ST300, the determination is positive, and the observation range restriction process proceeds to step ST302.

[0222] In step ST302, the CPU 22A acquires the three-dimensional area image 106 with spectator seat information from the NVM 22B by referring to the live view image received by the transmission / reception device 24 in step ST300. The CPU 22A also generates same-class area information based on the live view image received by the transmission / reception device 24 in step ST300 and the three-dimensional area image 106 with spectator seat information acquired from the NVM 22B. The CPU 22A then adds the same-class area information to the generated three-dimensional area image 106 with spectator seat information. After the processing of step ST302 is executed, the observation range restriction processing proceeds to step ST304.

[0223] In step ST304, the CPU 22A transmits the three-dimensional area image 106 with the same class area information and spectator seat information obtained in step ST304, i.e., the three-dimensional area image 106 with spectator seat information to which the same class area information has been added, to the user device 12, and then the observation range restriction process proceeds to step ST306.

[0224] In step ST306, the CPU 22A determines whether or not a condition for terminating the observation range restriction process (hereinafter referred to as the "observation range restriction process termination condition") has been satisfied. A first example of the observation range restriction process termination condition is that an instruction to terminate the observation range restriction process has been given to the information processing device 10 by an administrator of the information processing device 10 or the like. A second example of the observation range restriction process termination condition is that a fourth predetermined time (e.g., 10 hours) has elapsed since the execution of the observation range restriction process was started. A third example of the observation range restriction process termination condition is that the processing capacity of the CPU 22A has fallen below a reference level.

[0225] In step ST306, if the observation range restriction processing termination condition is not satisfied, the determination is negative and the observation range restriction processing proceeds to step ST300. If the observation range restriction processing termination condition is satisfied, the determination is positive and the observation range restriction processing ends.

[0226] As described above, in the information processing system 105, same-class area information is added to the three-dimensional region image 106 with spectator seat information, and the same-class area 110 in which the viewpoint position can be set is determined according to the class of the spectator seats 36B. The same-class area 110 is reflected in the reference image 108 (see FIG. 45). The reference image 108 reflecting the same-class area 110 is displayed on the display 18 of the user device 12. Therefore, with this configuration, the user 13 can visually recognize the area in which the viewpoint position can be set.

[0227] Furthermore, in the information processing system 105, the CPU 22A generates a three-dimensional area image 106 with spectator seat information to which same-class area information has been added. The same-class area information is information that can identify the same-class area 110, which is an area of ​​the same class as the area where the user 13 is watching. In other words, the three-dimensional area image 106 with spectator seat information to which same-class area information has been added is an image that can distinguish the same-class area 110 from other areas. Therefore, with this configuration, the user 13 can easily grasp the areas in which a viewpoint position can be set and the areas in which a viewpoint position cannot be set, compared to when an image is used in which the same-class area 110 cannot be distinguished from other areas.

[0228] Furthermore, in the information processing system 105, the reference image 108 (see FIG. 45) is an image based on the three-dimensional region image 106 with same-class area information and spectator seat information. That is, the reference image 108 is an image that allows the same-class area 110 to be identified, and is displayed on the display 18 of the user device 12. Therefore, with this configuration, the user 13 can visually recognize the area in which the viewpoint position can be set.

[0229] In the third embodiment, an example was described in which the range in which an observation position can be designated according to the class of the spectator seats 36B (in the example shown in FIG. 45 , the same-class area 110) is determined by the CPU 22A. However, the technology of the present disclosure is not limited to this. For example, the range in which an observation position can be designated may be determined by the CPU 22A according to the attributes of the user 13, such as the color of the team the user 13 is supporting, the color the user 13 likes most, the gender of the user 13, the age group of the user 13, and the clothing of the user 13, along with or instead of the class of the spectator seats 36B. In this case, for example, a three-dimensional area image with attribute information, in which information capable of identifying the attributes of the user 13 is added to the three-dimensional area image 32, may be stored in advance in the NVM 22B. Furthermore, the range in which an observation position can be designated according to the class of the spectator seats 36B is not limited to the same-class area and may be any range. Furthermore, the range in which observation positions can be specified according to the grade of the spectator seats 36B does not need to be divided by the range of the spectator seats, but may be divided, for example, within the soccer field 36A. More specifically, for example, the higher the grade of the spectator seats 36B, the closer to the goal the observation positions that can be specified may be.

[0230] Furthermore, in the above third embodiment, an example was given in which the reference image 108 is displayed on the display 18 of the user device 12, but the technology of the present disclosure is not limited to this, and the reference image 108 may be displayed on the display 74 of the HMD 68.

[0231] Furthermore, in each of the above embodiments, a soccer stadium 36 is used as an example, but the technology of the present disclosure is not limited to this, and may be any location where multiple imaging devices 30 can be installed, such as a baseball field, rugby field, curling court, athletics stadium, swimming pool, concert hall, outdoor music venue, theater venue, etc.

[0232] Furthermore, in each of the above embodiments, the computer 22 is exemplified, but the technology of the present disclosure is not limited to this. For example, a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 22. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the computer 22. The same applies to the computers 40 and 78.

[0233] Furthermore, in the above embodiment, the image generation processing program 34 and the observation range restriction processing program 104 (hereinafter, when it is not necessary to distinguish between them, they will be referred to as "programs") are stored in the NVM 22B, but the technology of the present disclosure is not limited to this, and as an example, as shown in FIG. 48, the programs may be stored in any portable storage medium 200 such as an SSD or USB memory, which is a non-transitory storage medium. In this case, the program stored in the storage medium 200 is installed in the computer 22, and the CPU 22A performs the image generation processing and the observation range restriction processing (hereinafter, when it is not necessary to distinguish between them) in accordance with the program. If no specific processing is required, the system executes the specific processing.

[0234] Alternatively, the program may be stored in a storage unit of another computer or server device connected to the computer 22 via a communication network (not shown), and the program may be downloaded to the information processing device 10 in response to a request from the information processing device 10. In this case, the specific process based on the downloaded program is executed by the CPU 22A of the computer 22.

[0235] In addition, while the above embodiments have exemplified the CPU 22A, the technology of the present disclosure is not limited to this, and a GPU may also be employed. Furthermore, multiple CPUs may also be employed instead of the CPU 22A. That is, a specific process may be executed by a single processor or multiple processors that are physically separated from each other.

[0236] The hardware resource for executing a specific process can be any of the following processors. As mentioned above, a processor can be, for example, a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process according to software, i.e., a program. Other processors can be, for example, dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing dedicated processes. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0237] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0238] Examples of systems configured with a single processor include, first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes specific processing. Second, a system in which a processor is used to realize the functions of an entire system, including multiple hardware resources that execute specific processing, on a single IC chip, as typified by SoCs. In this way, specific processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0239] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0240] Furthermore, the specific processing described above is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the present invention.

[0241] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0242] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0243] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; a memory embedded in or connected to the processor; The processor: acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area corresponding to an indication position indicated within the three-dimensional area to be observed or indicated within a reference image showing the state of the three-dimensional area when the three-dimensional area is observed from a reference position; the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the reference position is a position corresponding to the position of the imaging device, The subject image includes a second mark that can identify the pointing position. Information processing device.

2. The processor derives the coordinates based on an observation manner in which the three-dimensional area is observed and the indicated position. The information processing device according to claim 1 .

3. The observation mode is determined according to an observation position at which the three-dimensional area is observed. The information processing device according to claim 2 .

4. The processor determines an observation position designation range within which the observation position can be designated according to the attribute of the designation source. The information processing device according to claim 3 .

5. the processor acquires an image of an appearance within the three-dimensional area that shows an appearance within the three-dimensional area when the three-dimensional area is observed in the observation mode; The three-dimensional area interior aspect image is an image that is displayed in a manner that allows the observation position indication range within the three-dimensional area to be distinguished from a range other than the observation position indication range. The information processing device according to claim 4 .

6. The reference image is an image based on the image of the state within the three-dimensional region. The information processing device according to claim 5 .

7. The processor derives the coordinates based on a correspondence relationship between an image showing the state of the three-dimensional area when the three-dimensional area is observed in the observation state and a three-dimensional area image showing the three-dimensional area and whose position can be specified by the coordinates. The information processing device according to any one of claims 2 to 6.

8. The reference image is a virtual viewpoint image generated based on a plurality of images obtained by capturing images of the three-dimensional area using a plurality of imaging devices, or an image based on captured images obtained by capturing images of the three-dimensional area. The information processing device according to any one of claims 1 to 7.

9. The designated position designated in the reference image is a specific position in the virtual viewpoint image or the captured image. The information processing device according to claim 8 .

10. The reference image is an image including a first mark that can identify the pointing position within the reference image. The information processing device according to any one of claims 1 to 9.

11. When an object image showing an object present in the three-dimensional area when the three-dimensional area is observed from a position whose distance from the pointed position is within a threshold range or less is stored in a storage area, the processor acquires the object image instead of the subject image. The information processing device according to any one of claims 1 to 10.

12. The coordinates of the specific region within the three-dimensional region are coordinates that indicate a position higher than the actual position of the specific region within the three-dimensional region. The information processing device according to any one of claims 1 to 11.

13. the designated position designated within the three-dimensional region is a designated position on a first line extending from a viewpoint at which the three-dimensional region is observed toward a point of gaze, The designated position designated in the reference image is a designated position on a second line extending from the reference position to a specified point in the reference image. The information processing device according to any one of claims 1 to 12.

14. the designated position designated within the three-dimensional region is a position selected from at least one first candidate position; the designated position designated in the reference image is a position selected from at least one second candidate position; the processor associates, with the at least one first candidate position, a first reduced image obtained by reducing the object image when the three-dimensional area is observed from the first candidate position; A second reduced image obtained by reducing the subject image when the inside of the three-dimensional area is observed from the second candidate position is associated with the at least one second candidate position. The information processing device according to any one of claims 1 to 13.

15. The processor detects the pointing position based on a designated area image showing an area designated within the three-dimensional area. The information processing device according to any one of claims 1 to 14.

16. The subject image is a virtual viewpoint image generated based on a plurality of images obtained by capturing images of the three-dimensional area using a plurality of imaging devices. The information processing device according to any one of claims 1 to 15.

17. acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area that corresponds to a designated position within the three-dimensional area or a designated position within a reference image that shows the state of the three-dimensional area when the three-dimensional area is observed from a reference position, the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the reference position is a position corresponding to the position of the imaging device, The subject image includes a second mark that can identify the pointing position. Information processing methods.

18. A program for causing a computer to execute a process, acquiring a subject image showing a subject present in the three-dimensional area when the three-dimensional area is observed from a viewpoint position determined based on coordinates within the three-dimensional area that corresponds to a designated position within the three-dimensional area or a designated position within a reference image that shows the state of the three-dimensional area when the three-dimensional area is observed from a reference position, the reference image is an image obtained by an imaging device used by an observer who observes the subject image, the reference position is a position corresponding to the position of the imaging device, The subject image includes a second mark that can identify the pointing position. program.

Images