Display system and server

Abstract

A display system and a server, wherein the display system includes a display device and a remote terminal device. The display device includes a camera, an image recognition unit, and a display control unit. The image recognition unit recognizes real objects in an image captured by the camera, and recognizes the position of each recognized real object on the spatial coordinates. The display control unit displays a virtual object in an overlapping manner at a designated position on the spatial coordinates in the captured image. The image recognition unit recognizes the posture of the real object at the designated position of the virtual object in the captured image. The display control unit performs image processing on the virtual object based on the recognized posture.

Description

Technical Field

[0001] This specification discloses a display system for displaying an augmented reality (AR) image, and a server included in the system. Background Art

[0002] In the past, display devices using augmented reality technology have been known. For example, in Japanese Patent Application Laid-Open No. 2019-075125, an image processing of an imaginary (virtual) object (imaginary object) is performed by making a gesture (gesture) to the imaginary (virtual) object. For example, a folder icon (storage icon) as an imaginary object is projected on the display of a head-mounted display (HMD). By making a gesture such as pinching the three-dimensional icon with a finger, image processing such as expanding the folder icon is performed. Summary of the invention

[0003] Consider using the above-mentioned augmented reality technology to provide so-called AR game services such as tracking and capturing a virtual object that escapes. In such a case, if the escape path of the virtual object as the escapee is patterned, it will be difficult to maintain the appeal of the game. On the other hand, if an algorithm is created to complicate the escape path, the load on the system providing the AR game may become too large.

[0004] This specification discloses a display system and a server included in the system that can suppress the load of a system providing the AR game while maintaining the appeal of the AR game when providing the AR game.

[0005] In this specification, a display system is disclosed. The display system includes a display device and an operating device. The display device includes a camera, an image recognition unit, and a display control unit. The camera captures scenes in the real world. The image recognition unit recognizes real objects in the image captured by the camera, and recognizes the position of each recognized real object on the spatial coordinates. The display control unit causes a virtual object to be displayed overlappingly at a designated position on the spatial coordinates in the captured image. The image recognition unit recognizes the posture of the real object in the captured image at the designated position of the virtual object. The display control unit performs image processing on the virtual object based on the recognized posture. In addition, the operating device includes an input unit capable of inputting an operation instruction to change the designated position of the virtual object.

[0006] According to the above configuration, for example, the virtual object to be tracked in the AR game is operated by an operator who inputs an operation command to the operation device. This can avoid the patterning of the escape path of the virtual object without creating a complicated algorithm.

[0007] In addition, in the above configuration, the display system may further include a server capable of communicating with the display device and the operating device. In this case, the display device is arranged in a complex facility, and the complex facility is provided with a plurality of facilities having different specific themes. The server includes a storage unit and a transmission unit. The storage unit stores character images set as characters in each facility in association with location information of the complex facility. The transmission unit transmits the data of the character image together with the location information associated with the image to the display device. The display control unit selects the character image corresponding to the designated position as the image of the virtual object and displays it superimposed on the captured image.

[0008] According to the above configuration, it is possible to display a character image in accordance with the world view of a facility such as a theme park as an image of a virtual object.

[0009] In the above configuration, the gesture may be a capture action at a specified position. In this case, the display device sends a capture notification to the server when the image recognition unit recognizes the capture action of the real object. The server includes a reward calculation unit that determines a reward for the operator of the operating device based on the escape period from the start time of the operation of the virtual object to the recognition time of the capture action.

[0010] According to the above configuration, as an incentive for "allocating time to operate the virtual object", a reward is given to the operator of the virtual object.

[0011] In addition, in the above configuration, the display device may include an input unit for inputting registration of participation as a tracker of a tracking game of tracking a virtual object. In this case, the operating device includes an input unit for inputting registration of participation in the tracking game as an operator of the virtual object. The server includes a participant calculation unit that calculates the number of trackers and operators who are participating in the tracking game, and obtains the position of the virtual object operated by the operator and the display device held by the tracker in the complex facility. The reward calculation unit determines the reward for the operator based on the spatial density of the virtual object around the captured virtual object at the recognition time point of the capture action.

[0012] According to the above configuration, for example, different rewards can be given in a state where the virtual object space density is high, i.e., a state where it is difficult to capture, and a state where the virtual object space density is low, i.e., a state where it is easy to capture.

[0013] In the above configuration, the reward calculation unit may determine the reward for the operator of the operating device based on the spatial density of the trackers around the captured virtual object at the recognition time point of the capturing action.

[0014] According to the above configuration, different rewards can be given in a state where the tracker space density is high, that is, a state where it is easy to capture, and a state where the tracker space density is low, that is, a state where it is difficult to capture.

[0015] In addition, in this specification, a server is disclosed. The server can communicate with a display device and an operating device. The display device can execute a tracking game that displays a virtual object in a captured image obtained by capturing a scene in the real world and tracks the virtual object. The operating device has an input unit that can input operating instructions for the virtual object. The server has a storage unit and a sending unit. The storage unit stores position information of the display device during the period of participating in the tracking game and position information of the virtual object operated by the operating device. The sending unit sends position information of virtual objects around the display device to the display device based on predetermined position information of the display device.

[0016] According to the display system and the server disclosed in this specification, it is possible to suppress the load on the system providing the AR game and maintain the appeal of the AR game. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, in which like reference numerals represent like elements, and in which:

[0018] Figure 1 This is a diagram illustrating a complex amusement facility using the display system of this embodiment and houses in the distance therefrom.

[0019] Figure 2 This is a diagram for explaining a character display area in a complex amusement facility.

[0020] Figure 3 This is a diagram illustrating the hardware configuration of each device included in the display system of this embodiment.

[0021] Figure 4 This is a diagram illustrating the functional blocks of the server according to this embodiment.

[0022] Figure 5 A diagram showing a smartphone as an example of a display device.

[0023] Figure 6 This is a diagram illustrating the functional blocks of the display device according to this embodiment.

[0024] Figure 7 This is a diagram illustrating a functional block of a remote terminal device as an operating device according to the present embodiment.

[0025] Figure 8 This is a diagram illustrating the flow of a chaser in a chase game using the display system according to the present embodiment.

[0026] Fig. 9 This is a diagram illustrating a scene in a complex entertainment facility.

[0027] Fig.10 This is an example of photographing by a display device Fig. 9 Figure 1 shows an image captured when a scene is in progress.

[0028] Fig.11 This is a diagram (1 / 2) illustrating the process of capturing a virtual object.

[0029] Fig.12 FIG. 2 is a diagram (2 / 2) illustrating the process of capturing a virtual object.

[0030] Fig.13 This is a diagram illustrating the flow of a runner in a pursuit game using the display system of this embodiment.

[0031] Fig.14 The figure shows an example of a game image displayed on the display unit of the remote terminal device.

[0032] Fig.15 This is a diagram illustrating functional blocks of a server according to another example of the present embodiment.

[0033] Fig.16 This is a diagram illustrating functional blocks of a display device according to another example of the present embodiment.

[0034] Fig.17 This is a diagram illustrating functional blocks of a remote terminal device as another example of an operating device according to the present embodiment.

[0035] Fig.18 This is a diagram illustrating the flow of a chaser in a chase game using a display system according to another example of the present embodiment.

[0036] Fig.19 This is a diagram illustrating a flow of a runner in a pursuit game using a display system according to another example of the present embodiment.

[0037] Fig. 20 2 is a diagram illustrating another example of the display device according to the present embodiment, and is a diagram illustrating a head mounted display. DETAILED DESCRIPTION

[0038] exist Figure 1 , a complex amusement facility 10 and a house 92 using the display system of this embodiment are illustrated. The house 92 is located at a distance from the complex amusement facility 10, and an operator 94 who uses the display system is in the house 92. The display system is configured to include a display device 30 and a server 70 installed in the complex amusement facility 10, and a remote terminal device 100 (operating device) installed in the house 92.

[0039] <Composition of complex entertainment facilities>

[0040] A plurality of theme parks 14, 16, and 18 are provided in the complex amusement facility 10. A theme park is a facility that forms a world view based on a specific theme (topic), and based on the world view, comprehensively forms facilities, events, landscapes, etc., and presents a scenario. For example, the theme parks 14, 16, and 18 are connected by a communication path 20, and visitors to the complex amusement facility 10 can travel between the theme parks 14, 16, and 18 via the communication path 20.

[0041] Theme parks with different themes are provided in the complex amusement facility 10. For example, the complex amusement facility 10 includes an amusement park 14, an aquarium 16, and a zoo 18 as theme parks.

[0042] Each theme park 14, 16, 18 sets characters based on its own theme. The characters are set according to the theme and world view of each theme park 14, 16, 18. For example, in the amusement park 14, characters such as clowns and go-karts are set. For example, in the aquarium 16, characters such as fish and sharks are set. And, in the zoo 18, characters such as monkeys, lions, and pandas are set.

[0043] As described later, these character images (hereinafter appropriately referred to as character images) are used as images of escapees in a pursuit game that is an AR game service that can be utilized in the complex amusement facility 10 .

[0044] The tracking game is a game similar to "hide and seek" that can be experienced by the attendees of the complex amusement facility 10 and uses augmented reality technology. As described later, in this tracking game, a virtual image of the escapee as the tracking object is superimposed on the captured image in the complex amusement facility 10, and the superimposed image (AR image) is displayed on the display device 30. The attendees as the chaser chase the virtual image in order to capture the virtual image. In addition, in the display system of this embodiment, the operation of the virtual image as the escapee is performed by a remote terminal device 100 away from the complex amusement facility 10.

[0045] Visitors can not only walk through the theme parks 14, 16, 18 and the connecting passage 20, but also walk through the facility 10. For example, the outside of the connecting passage 20 is a lawn for walking. In the tracking game, the tracking area (tracking field) is set to be, for example, the entire area of ​​the complex entertainment facility 10. In addition, in the tracking game, Figure 2As shown in the example, the area within the complex amusement facility 10 is virtually divided into a plurality of areas. For example, these areas are divided into three areas: an amusement park area 14A including an amusement park 14, an aquarium area 16A including an aquarium 16, and a zoo area 18A including a zoo 18. In addition, the boundaries of these areas are imaginary, and no physical boundary units such as fences are provided. Visitors to the complex amusement facility 10 can freely move to each area.

[0046] In the tracking game, the images of virtual objects (character images) set to be displayed in each area 14A to 18A are different. For example, in the amusement park area 14A, the character images set for the amusement park 14 are displayed as virtual object images. Similarly, in the aquarium area 16A, the character images set for the aquarium 16 are displayed as virtual object images, and in the zoo area 18A, the character images set for the zoo 18 are displayed as virtual object images.

[0047] The character image data (character image data) of the theme parks 14, 16, 18 are stored in the server 70 (see Figure 4 ) for each theme park. The server 70 is installed in, for example, a building (not shown) for management provided in the complex amusement facility 10. Details of the character image data stored in the character storage unit 82 for each theme park will be described later.

[0048] return Figure 1 A beacon transmitter 22 is provided in the complex entertainment facility 10. For example, a plurality of transmitters 22 are provided at equal intervals. As described later, the beacon receiver 37 (see FIG. 1 ) of the display device 30 is used to transmit the beacon to the display device 30. Figure 3 ) receives the signal from the transmitter 22, thereby being able to obtain the current position of the display device 30.

[0049] A remote terminal device 100 is arranged in a house 92 far away from the complex amusement facility 10. As described in detail below, in the display system of this embodiment, in the tracking game which is an AR game, a virtual object (character) as a runner is operated by the remote terminal device 100. That is, the attendees of the complex amusement facility 10 participate in the tracking game as a tracker, and on the other hand, the virtual object as the tracking object is operated by the operator 94 who is far away from the complex amusement facility 10 by inputting an operation instruction to the remote terminal device 100.

[0050] <Display system configuration>

[0051] exist Figure 32 shows an example of the hardware configuration of the devices included in the display system of the present embodiment. The display system of the present embodiment is configured to include a display device 30 , a server 70 , and a remote terminal device 100 .

[0052] <Server Configuration>

[0053] Reference Figure 3 The server 70 is composed of a computer, for example, and is disposed in the complex entertainment facility 10 (see Figure 1 ) in a management building. In addition, the server 70 is wirelessly connected to the display device 30 via a communication unit such as a wireless LAN. Furthermore, the server 70 can communicate with the remote terminal device 100 via a communication unit such as the Internet.

[0054] The server 70 includes an input unit 71 such as a keyboard and a mouse, a CPU 72 of a computing device, and a display unit 73 such as a display. In addition, the server 70 includes a ROM 74, a RAM 75, and a hard disk drive 76 (HDD) as a storage device. The server 70 also includes an input / output controller 77 for managing the input / output of information. These components are connected to an internal bus 78.

[0055] exist Figure 4 2 shows an example of a functional block of the server 70. The functional block is configured by, for example, the CPU 72 executing a program stored in the ROM 74, the HDD 76, or a computer-readable non-transitory storage medium such as a DVD.

[0056] The server 70 includes a receiving unit 83, a participant calculating unit 84, a reward calculating unit 85, a 3D model extracting unit 86, and a transmitting unit 87 as functional units. The server 70 also includes a tracking game participant storage unit 80, a facility map storage unit 81, and a character storage unit 82 for each theme park as storage units.

[0057] The tracking game participant storage unit 80 stores information about participants of the AR game, i.e., the tracking game. Participants include trackers and escapers. Trackers include attendees of the complex amusement facility 10 who have applied for registration as trackers of the tracking game from the display device 30 held by the attendees. On the other hand, escapers include those who have applied for registration as escapers of the tracking game from the remote terminal device 100.

[0058] The tracking game participant storage unit 80 stores information about the tracker and the escapee who are participating in the tracking game. For example, the tracker information includes the identification mark and position information of the display device 30 held by the tracker. In addition, the escapee information includes the operator 94 (see Figure 1) and the identification mark of the remote terminal device 100 operated by the remote terminal device 100 and the position information of the virtual object operated by the remote terminal device 100.

[0059] The facility map storage unit 81 stores park map data as map information within the complex amusement facility 10. For example, location information of the communication paths 20 and facilities within the complex amusement facility 10 is stored as park map data. Figure 1 ) etc. For example, the no-entry area is set as a no-movement area for the escapee.

[0060] Specifically, in the facility map storage unit 81, the map data of the complex amusement facility 10 is stored as park map data in a state associated with the position information. The position information is the position information on the so-called world coordinate system. The position information on the world coordinate system includes latitude, longitude, and altitude information that can be obtained using the GPS function, and spatial coordinate information that can be obtained using the beacon function.

[0061] In addition, the facility map storage unit 81 stores Figure 2 The data of the areas 14A, 16A, and 18A in the illustrated tracking game are, for example, set on the park map data the boundary lines of the areas 14A, 16A, and 18A, and different identification marks are given to the areas divided by the boundary lines according to the areas 14A, 16A, and 18A.

[0062] The image of the escapee displayed as a virtual object image on the display device 30, that is, the data of the character image, is stored in the character storage unit 82 for each theme park. As the characters in the theme parks 14, 16, and 18, the characters are set according to their respective worldviews. For example, according to the theme parks 14, 16, and 18, a plurality of types of character image data are stored in the character storage unit 82 for each theme park. For example, for one theme park, more than 10 and less than 100 types of character image data are stored in the character storage unit 82 for each theme park. Identification marks are given to the above-mentioned character image data respectively according to the theme parks 14, 16, and 18, that is, according to the areas 14A, 16A, and 18A. In addition, each character image data is given a unique identification mark.

[0063] In addition, character image data as a virtual object and position information associated with the data are stored in the character storage unit 82 for each theme park. The position information associated with the character image data includes area information within the complex entertainment facility 10 that allows the image data to be displayed. Hereinafter, the character image data and the position information data associated with the character image data are collectively described as "virtual object data" as appropriate.

[0064] For example, Figure 2 As shown in the example, the character image 15, which is assigned an identification mark corresponding to the amusement park 14 and whose display area is set to the amusement park area 14A, includes an image of a clown performing a ball-stepping performance. In addition, the character image 17, which is assigned an identification mark corresponding to the aquarium 16 and whose display area is set to the aquarium area 16A, includes an image of a swimming fish. Similarly, the character image 19, which is assigned an identification mark corresponding to the zoo 18 and whose display area is set to the zoo area 18A, includes an image of a running monkey.

[0065] The image data of the character may be 3D model data. The 3D model data, for example, includes 3D image data of the character, and the image data includes shape data, texture data, and motion data. Furthermore, the image data also includes data of an effect image when the virtual object is captured.

[0066] In addition, Figure 2 In order to make the illustration clear, outline images are shown as the character images 15, 17, and 19, but the character images 15, 17, and 19 are not limited to this form, and 3D images of the character images 15, 17, and 19 may be displayed.

[0067] The participant calculation unit 84 calculates the number of participants who are participating in the tracking game, etc. Participants include trackers and escapers. In addition, the participant calculation unit 84 obtains the position information of the trackers who are participating in the tracking game, or more specifically, the display device 30 held by the trackers, within the complex amusement facility 10. In addition, the participant calculation unit 84 obtains the position information of the escapers (virtual objects) operated by the operators who are participating in the tracking game within the complex amusement facility 10. In addition, for example, the participant calculation unit 84 refers to the tracking game participant storage unit 80 to calculate the tracker space density and the escaper space density (virtual object space density) described later.

[0068] In addition, the participant calculation unit 84 provides the display device 30 that is participating in the tracking game (as a tracker) with the position information of the virtual object (escaper) around the position based on the position information of the display device 30. In addition, the participant calculation unit 84 provides the remote terminal device 100 that is participating in the tracking game (as a runner) with the park map data and the position information of the runner and the tracker.

[0069] The 3D model extraction unit 86 extracts image data of a virtual object, that is, character image data, from the character storage unit 82 for each theme park, and transmits the extracted image data to the display device 30 and the remote terminal device 100 via the transmission unit 87 .

[0070] The 3D model extraction unit 86 may send the character image data of all the regions 14A, 16A, and 18A to the display device 30 and the remote terminal device 100. For example, any one type of character image data may be sent to the display device 30 and the remote terminal device 100 for each region 14A, 16A, and 18A.

[0071] The reward calculation unit 85 calculates the reward given to the operator who controls the escapee in the pursuit game. For example, the reward is calculated based on the escape time. In addition, other elements of reward calculation include the spatial density of escapees and the spatial density of pursuers around the time of capture. The reward can be, for example, a discount service for the entrance fee to the complex entertainment facility 10. The calculation of the reward will be described later.

[0072] <Configuration of Display Device 30>

[0073] Reference Figure 1 The display device 30 is disposed in the complex amusement facility 10 and is used by attendees (users) of the facility 10. The display device 30 can display a virtual reality image obtained by superimposing an image of a virtual object on a scene in the real world.

[0074] The display device 30 may be a portable device, for example, a smartphone including a camera and a display unit, or a glasses-type head mounted display (HMD).

[0075] From the perspective of how the real-world scene is displayed, the display device 30 can be divided into a video see-through display (VST display) and an optical see-through display (OST display). In a VST display, a camera or other image capturing device captures the real-world scene and displays the captured image on a display unit. On the other hand, an OST display visually recognizes the real-world scene through a see-through display unit such as a half mirror and projects a virtual object on the display unit.

[0076] The above-mentioned smart phone, etc., is equipped with a camera 35 (see Figure 3 ) is classified as a VST display. In addition, the head mounted display (HMD) described above uses the lenses of glasses as a display unit to visually recognize scenes in the real world, and is therefore classified as an OST display.

[0077] In the following implementation, Figure 5 As illustrated, a portable VST display type smartphone is shown as an example of the display device 30. The smartphone may be a property of a visitor of the complex amusement facility 10 or a rental item such as a tablet terminal lent to a visitor of the complex amusement facility 10.

[0078] exist Figure 3 , the hardware configuration of the display device 30 is illustrated. The display device 30 includes a CPU 31 (central processing unit), an acceleration sensor 32, a gyro sensor 33, a geomagnetic sensor 34, a camera 35, a GPS receiver 36, and a beacon receiver 37. The display device 30 also includes an input / output controller 39, a system memory 40, a storage device 41, a GPU (Graphics Processing Unit) 42, a frame memory 43, a RAMDAC (Random Access Memory Digital-to-Analog Converter) 44, a display control unit 45, a display unit 46, and an input unit 47.

[0079] The system memory 40 is a storage device used by an operating system (OS) executed by the CPU 31. The storage device 41 is an external storage device, and stores, for example, a program for displaying a virtual reality image (AR image) described later.

[0080] The direction and orientation of the display device 30 can be estimated by the acceleration sensor 32, the gyro sensor 33, and the geomagnetic sensor 34. The acceleration sensor 32 measures the acceleration of the display device 30. Figure 5 As shown in the example, the acceleration sensor 32 can measure accelerations in three orthogonal axial directions. That is, the accelerations in the axial directions of the X-axis and Y-axis which are parallel to the display surface of the display unit 46 and orthogonal to each other, and the Z-axis which is orthogonal to the display surface of the display unit 46 are measured by the acceleration sensor 32. The acceleration sensor 32, the gyro sensor 33, and the geomagnetic sensor 34 are composed of so-called micromachines such as MEMS (MicroElectro Mechanical Systems).

[0081] The gyro sensor 33 measures the angular velocity of the display device 30. Figure 5 As shown in the example, the gyro sensor 33 measures rotations around three orthogonal axes. That is, the pitch angle, which is rotation around the X-axis, the roll angle, which is rotation around the Y-axis, and the azimuth angle (also called yaw angle), which is rotation around the Z-axis, are measured by the gyro sensor 33. In addition, the geomagnetic sensor 34 detects the tilt of the display device 30 relative to the magnetic north.

[0082] The camera 35 is, for example, a camera device mounted on a smartphone, and can capture scenes of the real world in static and dynamic images. The camera 35 is, for example, configured to include a capturing device such as a CMOS or a CCD. Furthermore, the camera 35 may be a so-called RGB-D camera that has a function of measuring the distance from the camera 35 in addition to capturing the real world. As a function of measuring the distance, for example, in the camera 35, in addition to the above-mentioned capturing device, a distance measuring mechanism using infrared rays is also provided.

[0083] The GPU 42 is a computing device for image processing, and mainly operates when performing image recognition described later. The frame memory 43 is a storage device for storing images captured by the camera 35 and then processed by the GPU 42. The RAMDAC 44 converts the image data stored in the frame memory 43 into an analog signal for the display unit 46 as an analog display.

[0084] The GPS receiver 36 receives data from the GPS satellites 24 (see Figure 1 ) receives a positioning signal, namely a GPS signal. The GPS signal includes position coordinate information of latitude, longitude, and altitude. The beacon receiver 37 receives a position signal output from a transmitter 22 of a beacon installed in the complex entertainment facility 10.

[0085] Here, the position estimation functions of the GPS receiver 36 and the beacon receiver 37 overlap. Therefore, only one of the GPS receiver 36 and the beacon receiver 37 may be provided in the display device 30.

[0086] The input unit 47 can input an activation command or a shooting command to the imager 35. For example, the input unit 47 may be a touch panel integrated with the display unit 46.

[0087] exist Figure 6 4 shows the functional blocks of the display device 30. The functional blocks are formed by, for example, the CPU 31 and the GPU 42 executing a program stored in the system memory 40 and the storage device 41. Alternatively, the functional blocks are formed by the CPU 31 and the GPU 42 executing a program stored in a non-temporary storage medium readable by a computer, such as a DVD or a hard disk of a computer. Figure 6 Functional blocks illustrated.

[0088] exist Figure 6 in Figure 3 The configuration of the display device 30 is shown in a state where a part of the hardware configuration illustrated in the figure is mixed with the functional blocks. Figure 6 The image capturing device 35 , the display control unit 45 , the input unit 47 , and the display unit 46 are exemplified as the hardware configuration.

[0089] In addition, the display device 30 includes a position information acquisition unit 50, a transmission unit 52, a reception unit 55, a direction estimation unit 56, and an image recognition unit 58 as functional blocks. In addition, the display device 30 includes a virtual object data storage unit 57 and a learned model storage unit 59 as storage units. These functional blocks are composed of the CPU 31, the system memory 40, the storage device 41, the GPU 42, and the frame memory 43.

[0090] The position information acquisition unit 50 obtains Figure 3 At least one of the GPS receiver 36 and the beacon receiver 37 obtains information on the current position of the display device 30. This position information is the position information of the so-called world coordinate system, and in the case of a GPS signal, it is included in the position information on the space coordinate system composed of latitude, longitude and altitude information. In addition, in the case where the received position information is the position information obtained from the beacon signal, as the position information of the world coordinate system, for example, the x-coordinate, y-coordinate and z-coordinate on the space coordinate system with an arbitrary location of the complex entertainment facility 10 as the origin are included in the position information. In addition, here, the x-coordinate and the y-coordinate can be orthogonal coordinate axes on the horizontal plane, and the z-coordinate can be a vertical axis.

[0091] The orientation and azimuth estimation unit 56 estimates the orientation and azimuth of the display device 30. The orientation and azimuth estimation unit 56 is configured to include the acceleration sensor 32 (see Figure 3 ), gyro sensor 33 and geomagnetic sensor 34. The geomagnetic sensor 34 can detect the inclination of the display device 30 relative to magnetic north. In addition, the acceleration sensor 32 and the gyro sensor 33 can detect Figure 5 As a result, for example, the direction of the optical axis of the camera 35 with respect to magnetic north can be obtained.

[0092] The image recognition unit 58 receives the captured image data from the camera 35 and performs image recognition. Image recognition includes recognition of real objects in the captured image and estimation of the distance between the recognized real objects and the display device 30. When performing such image recognition, for example, as described above, the captured image data includes, in addition to color image data obtained by capturing a scene in the real world, distance data of each pixel of the color image data from the camera 35.

[0093] The image recognition unit 58 uses the learning model for image recognition stored in the learned model storage unit 59 to recognize real objects in the captured image. The learned model storage unit 59 stores, for example, a neural network for image recognition that has been learned (already learned) by an external server or the like. For example, segmented and annotated data of each object in the image, which includes image data of the exterior of the complex amusement facility 10, is prepared as training data. Using the training data, a multi-layer neural network that has been machine-learned through teacher learning is formed and stored in the learned model storage unit 59. The neural network may be, for example, a convolutional neural network (CNN).

[0094] As described above, the camera 35 is an RGB-D camera provided with a distance measurement function, and can obtain the distance between each pixel in the captured image and the camera 35. The distance between the real object in the captured image recognized by image recognition and the camera 35 can be calculated using the distance. Thus, for each real object in the captured image, position information in a so-called local coordinate system with a predetermined point (e.g., focus) of the camera 35 as the origin can be obtained.

[0095] Furthermore, based on the orientation of the camera 35 estimated by the orientation estimation unit 56 and the position of the display device 30 on the world coordinate system obtained by the position information acquisition unit 50, the position of the real object in the captured image on the world coordinate system (position on the spatial coordinate system) can be determined.

[0096] In addition, although real objects located far away in the depth direction are also captured in the captured image, in the pursuit game, the escapee (character) near the pursuer is usually captured. Therefore, for example, when the distance information obtained for each pixel of the captured image exceeds a predetermined interval distance (e.g., 100m), the position of the pixel in the world coordinate system may not be obtained.

[0097] The virtual object data storage unit 57 stores image data of a virtual object, i.e., a character, representing the escapee. The character image data of all the regions 14A, 16A, and 18A can be stored in the virtual object data storage unit 57. For example, the virtual object data storage unit 57 stores any one type of character image data for each region 14A, 16A, and 18A. In addition, position information is associated with these character image data. For example, the range of position coordinates that can be displayed as a virtual object is associated with each character image data.

[0098] The display control unit 45 can generate an augmented reality image (AR image) obtained by overlapping the image of a virtual object with a scene in the real world, and display it on the display unit 46. For example, the display control unit 45 overlaps the image of a virtual object, i.e., a character, with a captured image obtained by capturing a scene in the real world by the camera 35, and displays the overlapped image on the display unit 46. The display unit 46 may be, for example, a liquid crystal display or an organic EL display.

[0099] As described later Fig.10 In this way, the display control unit 45 generates an augmented reality image in which the image of the virtual object that is the escapee in the tracking game is overlapped with the captured image of the camera 35. The display control unit 45 obtains the position information (designated position) of the escapee (virtual object) from the server 70. Furthermore, the display control unit 45 overlaps the image of the escapee, that is, the character image, at the designated position in the captured image. At this time, as the overlapping character image, the character image corresponding to the designated position, that is, the character image set to the area 14A, 16A, 18A included in the designated position is selected.

[0100] Furthermore, in a pursuit game, the escapee (character) near the pursuer is usually captured, so when the designated position exceeds a predetermined interval distance (e.g., 100 m), the character image of the escapee corresponding to the designated position may not be superimposed on the captured image.

[0101] <Composition of remote terminal device>

[0102] The remote terminal device 100 is operated by an operator 94 (see Figure 1 ) operating device. Figure 1 The middle figure shows one remote terminal device 100 and one operator 94, but a plurality of remote terminal devices 100 can communicate with the server 70, so that a plurality of operators 94 can participate in the pursuit game as escapees.

[0103] The remote terminal device 100 (operating device) may be, for example, a computer. Figure 3 The remote terminal device 100 includes a CPU 101 as a computing device, a system memory 102 as a storage device, and a storage device 104.

[0104] The remote terminal 100 also includes a display unit 106 and a display control unit 105 that controls an image displayed on the display unit 106. The remote terminal 100 also includes an input / output controller 103 that manages input / output of information. The remote terminal 100 can communicate with the server 70 via a communication unit such as the Internet.

[0105] The remote terminal device 100 also includes an input unit 107 for inputting information. The input unit 107 is a device such as a mouse or a keyboard that can input operation instructions for the escapee. As described later, the operation instructions for the escapee include instructions for setting the position (specified position) of a virtual object as the escapee.

[0106] exist Figure 7 The functional blocks of the remote terminal device 100 are shown in FIG. Figure 3 The remote terminal device 100 includes a transmission unit 110, a reception unit 111, a virtual object data storage unit 112, and a facility map storage unit 113 as functional blocks.

[0107] The virtual object data storage unit 112 stores image data of a virtual object, i.e., a character, of the escapee. The character image data is associated with position information. For example, the range of position coordinates on the complex amusement facility 10 that is allowed to be displayed as an image representing the virtual object of the escapee is associated with the character image data and stored in the virtual object data storage unit 112.

[0108] The facility map storage unit 113 stores park map data of the complex amusement facility 10. The park map data may be stored in the server 70 (see Figure 4 )'s facility map storage unit 81 has the same data as the park map data.

[0109] As described later Fig.14 In this way, the display control unit 105 generates a game image in which the images of the escapee and the pursuer in the complex amusement facility 10 are superimposed on the image of the park map, that is, the overhead view of the complex amusement facility 10. For example, the current position information of the escapee and the pursuer is provided by the server 70. The display control unit 105 superimposes the images of the escapee and the pursuer on the current positions provided by the server 70 on the park map.

[0110] <Tracking Game Tracker Side Process>

[0111] exist Figure 8 FIG. 2 illustrates an example of a process on the tracker side of an AR game, that is, a tracking game, using the display system of this embodiment. In the process on the tracker side, the server 70 and the display device 30 are the main bodies that exclusively execute the steps. Figure 8 In each step of the process, the step executed by the server 70 is indicated by (S). In addition, the step executed by the display device 30 is indicated by (D).

[0112] Reference Figure 4 , Figure 6 , Figure 8The tracker process is started when a registration application for participating in the tracking game is input to the input unit 47 of the display device 30 disposed in the complex amusement facility 10. Upon receiving the input of the registration application, the position information acquisition unit 50 of the display device 30 acquires the current position of the display device 30. The current position is the position coordinate on the world coordinate system. In addition, upon receiving the input of the registration application, the camera 35 is started and captures the scene in the complex amusement facility 10.

[0113] The position information acquired by the position information acquisition unit 50 and the participation registration information input from the input unit 47 are transmitted from the transmission unit 52 to the server 70 (S10). The reception unit 83 of the server 70 transmits the position information of the display device 30 and the participation registration information to the participant calculation unit 84. In addition, the reception unit 83 transmits the position information of the display device 30 to the 3D model extraction unit 86.

[0114] The 3D model extraction unit 86 extracts virtual object data from the character storage unit 82 for each theme park. As described above, the virtual object data includes character image data and position information data (display area data) associated therewith. The virtual object data is transmitted from the transmission unit 87 to the display device 30 (S12).

[0115] The participant calculation unit 84 stores the location information and participation registration information received from the receiving unit 83 in the tracking game participant storage unit 80. In addition, the participant calculation unit 84 obtains information of participants participating in the tracking game from the tracking game participant storage unit 80. This information includes the location information of the runner. The location information of the runner may be the location information of all runners who are participating in the tracking game. Alternatively, the location information of the runner may be the location information of the runner within a predetermined distance from the display device 30. For example, the participant calculation unit 84 obtains the location information of the runner in an area within a radius of 100m from the display device 30. The location information of the runner may be the location coordinate information of the world coordinate system. The participant calculation unit 84 sends the obtained location information of the runner to the display device 30 (S14).

[0116] The image recognition unit 58 of the display device 30 performs image recognition on the image captured by the camera 35, and performs segmentation and annotation of the real object in the captured image. In addition, the image recognition unit 58 calculates the distance between the recognized real object and the display device 30 based on the distance measurement function of the camera 35 (S16).

[0117] Next, the image recognition unit 58 obtains the direction and orientation of the optical axis of the camera 35 through the orientation estimation unit 56 (S18). For example, the image recognition unit 58 obtains the orientation (angle) of the optical axis of the camera 35 relative to the magnetic north and the rotation angle (orientation) around three orthogonal axes including the optical axis.

[0118] Furthermore, the image recognition unit 58 obtains the position of the display device 30 in the world coordinate system through the position information acquisition unit 50 (S20). In addition, the image recognition unit 58 obtains the position of the real object in the captured image in the world coordinate system (S22). The captured image, the real object information recognized in the image, and the position information of the real object in the world coordinate system are sent to the display control unit 45.

[0119] The display control unit 45 obtains the position coordinates of the escapee included in the captured image (S24). For example, the display control unit 45 determines whether the position coordinates of the plurality of escapees sent from the server 70 are included in the captured image.

[0120] Furthermore, the display control unit 45 causes the imaginary object image, i.e., the character image, to overlap the position (specified position) of the escapee in the captured image (S26). In addition, the character image to be overlapped at this time is selected based on the position information of the escapee. For example, a character image set corresponding to the area 14A, 16A, or 18A including the position of the escapee is selected and overlapped with the captured image.

[0121] exist Fig. 9 The scene in the complex entertainment facility 10 is shown in FIG. Fig.10 3 shows an example in which the scene is photographed by the camera 35 of the display device 30 and the photographed image is displayed on the display unit 46. In this example, the photographed image includes the location of the escapee, that is, the designated location (X1, Y1, Z1), and the character image 19 is superimposed on the designated location. In addition, in this example, it is shown that the designated location (X1, Y1, Z1) is included in the zoo area 18A (see Figure 2 ) example, the character image 19 of the monkey set for the area is superimposed on the captured image.

[0122] Then, the image recognition unit 58 performs capture determination of the virtual object (S28). Fig.11 , when the display unit 46 displays the character image 19 , the hand 120 of the pursuer enters the photographing field of the camera 35 , that is, the hand 120 of the pursuer enters the opposite side of the display unit 46 .

[0123] Furthermore, the hand 120 of the pursuer is photographed by the camera 35 and recognized by the image recognition unit 58. Fig.12 As illustrated, when the position of the hand 121 of the tracker, which is the real object in the captured image, matches the designated position of the virtual object and a predetermined gesture is made by the hand 121, the image recognition unit 58 recognizes the gesture. Fig.12As shown in the example, the image recognition unit 58 determines that the imaginary object as the escapee is captured by the pursuer by recognizing the gesture. In short, the gesture is an action (capturing action) such as capturing the imaginary object image, i.e., the character image, with the hand 120 on the display unit 46. When the image recognition unit 58 recognizes the capturing action of the character image, the image recognition unit 58 sends a capture notification to the server 70 via the sending unit 52 (S30). The capture notification includes the identification mark of the pursuer (display device 30), the location where the capture is performed, and the identification mark of the captured escapee.

[0124] In addition, when the image recognition unit 58 determines that the escapee has been captured, the display control unit 45 performs image processing when the virtual object is captured. Fig.12 As illustrated, the display control unit 45 causes the display unit 46 to display a radial line graph 122 indicating that the character image 19 has disappeared.

[0125] The capture notification is sent to the reward calculation unit 85 of the server 70. The reward calculation unit 85 gives the chaser a reward for the capture (S32). For example, as a reward, a score is sent to the display device 30. The score is displayed on the display unit 46 of the display device 30. By capturing the escapee, the pursuit game for the chaser who has performed the capture ends.

[0126] On the other hand, when the image recognition unit 58 determines in step S28 that the escapee (virtual object) has not been captured by the pursuer, the process returns to step S14 and acquires the position coordinates of the escapee from the server 70 again.

[0127] In this way, in the pursuit game, the pursuer can enjoy the game of chasing the imaginary object (escaper) displayed on the display unit 46 of the display device 30. And, as the image of the imaginary object displayed on the display unit 46, the character image corresponding to the area 14A, 16A, 18A where the imaginary object is located is selected. In addition, for example, when the position coordinates of the imaginary object cross the area, the displayed character image changes. By performing such display control, it is possible to perform a scene presentation according to the world view of each theme park 14, 16, 18 of the complex amusement facility 10.

[0128] <Tracking the game escapee side process>

[0129] exist Fig.13 The process of tracking the escapee of the game is illustrated in FIG. The steps of this process are exclusively executed by the server 70 and the remote terminal device 100. Fig.13 In FIG. 1 , the steps executed by the server 70 are indicated by (S), and the steps executed by the remote terminal device 100 are indicated by (T). Figure 4 , Figure 7 , Fig.13When a registration application for participation in the tracking game is input to the input unit 107 of the remote terminal device 100, Fig.13 The process (the escapee side process) is started.

[0130] The remote terminal device 100 transmits the participation registration information to the server 70 (S50). The participation registration information includes the remote terminal device 100 or its operator 94 (refer to Figure 1 The receiving unit 83 of the server 70 that has received the participation registration information transmits the information to the participant calculation unit 84. The participant calculation unit 84 stores the participation registration information in the tracking game participant storage unit 80.

[0131] Furthermore, the participant calculation unit 84 obtains the position information of all the trackers who are currently participating in the tracking game from the tracking game participant storage unit 80 (S52). Next, the participant calculation unit 84 sets the vicinity of the tracker as the initial position of the runner (S54). In the case where multiple trackers are participating in the tracking game, a tracker is randomly selected and its vicinity is set as the initial position of the runner. The vicinity of the tracker refers to, for example, a range within a radius of 1m from the tracker's position.

[0132] Furthermore, the participant calculation unit 84 obtains the park map data from the facility map storage unit 81 and transmits the data to the remote terminal device 100 via the transmission unit 87 (S56). In addition, the 3D model extraction unit 86 receives the participant registration information and transmits the virtual object data to the remote terminal device 100.

[0133] Next, the participant calculation unit 84 sends the position information of the escaper and the pursuer to the remote terminal device 100 (S58). The pursuer's position information includes the position information of all pursuers who are participating in the pursuit game in the complex amusement facility 10. Similarly, the escaper's position information includes the position information of all escapers who are participating in the pursuit game in the complex amusement facility 10. In addition, the escaper's initial position information set in step S54, that is, the escaper's own initial position information, is also included in the escaper's position information.

[0134] In the remote terminal device 100, the virtual object data is stored in the virtual object data storage unit 112. In addition, the park map data is stored in the facility map storage unit 113. Then, the display control unit 105 generates a park map image with virtual object images of the pursuer and the escapee including the pursuer superimposed thereon as a game image, and displays it on the display unit 106 (S60).

[0135] exist Fig.14. In this game image, character images 15, 17, 19 overlap with the position coordinates of the escapee on the park map image. The escapee's character image 15, 17, 19 is selected according to which area 14A, 16A, 18A the escapee's position is included in. In contrast, the pursuer's character image 125 can be set in the same manner regardless of the area 14A, 16A, 18A.

[0136] In addition, the character image 19A representing the escapee operated by the operator via the remote terminal device 100 may be subjected to image processing such as adding shading so as to be distinguishable from other character images.

[0137] In response to displaying on the display unit 106 Fig.14 Such a game image, the operator 94 (refer to Figure 1 ) inputs an operation instruction to the input unit 107 in such a manner that the character image 19A of the escaper as the operation target can escape from the pursuer (S62). That is, the operator 94 inputs an operation instruction to the input unit 107 to set a position farther from the pursuer than the current position as the designated position of the escaper (virtual object).

[0138] The input operation command is sent to the server 70. Here, the reward calculation unit 85 determines whether a capture notification for the character image 19A is sent from the display device 30 as the tracker (S64). If a capture notification for the character image 19A is not sent, the process returns to step S58.

[0139] On the other hand, in step S64 , when the capture notification is transmitted to the server 70 for the character image 19A, the bonus calculation unit 85 calculates the bonus.

[0140] First, the reward calculation unit 85 calculates the score coefficient k1 based on the escape period of the character image 19A (S66). The escape period, for example, refers to the period from the start time of the operation of the imaginary object, i.e., the character image 19A, to the recognition time of the capture action, i.e., the output time of the capture notification. The score coefficient k1 is set to a high value in proportion to the escape period.

[0141] Next, the reward calculation unit 85 calculates the score coefficient k2 (S68) based on the escapee density at the capture time point of the character image 19A. The escapee density can also be called the virtual object space density, which represents the space density around the virtual object image, i.e., the character image 19A. For example, the space density of the virtual object (escapee) within a radius of 5m from the position coordinates of the captured character image 19A is calculated as the escapee density. In addition, the space density of the virtual object (escapee) at the recognition time point of the capture action, i.e., the output time point of the capture notification, is calculated as the escapee density.

[0142] The higher the escapee density is, the more difficult it is to capture the escapee. Therefore, the score coefficient k2 is set to be lower as the escapee density is higher. For example, the score coefficient k2 is set to be inversely proportional to the escapee density.

[0143] Next, the reward calculation unit 85 calculates the score coefficient k3 based on the density of the trackers at the time point of capturing the character image 19A (S70). The tracker density can also be called the tracker spatial density, which indicates the spatial density of the trackers around the character image 19A. For example, the spatial density of the display device 30 within a radius of 5m from the position coordinates of the character image 19A is calculated as the tracker density. In addition, the spatial density of the display device 30 at the time point of recognizing the capture action of the character image 19A, that is, the output time point of the capture notification, is calculated as the tracker density.

[0144] The higher the density of the pursuers, the easier it is to catch the escapee. Therefore, the higher the density of the pursuers, the higher the score coefficient k3 is set. For example, the score coefficient k3 is set to be proportional to the density of the pursuers.

[0145] Furthermore, the reward calculation unit 85 calculates the score obtained by multiplying the predetermined basic points by the score coefficients k1, k2, and k3 and gives it to the remote terminal device 100 (S72). Figure 1 ) When arriving at the complex entertainment facility 10, they can enjoy discount services corresponding to the points assigned.

[0146] According to the display system described above, the escape route of the escapee in the tracking game is determined by the operator of the remote terminal device 100. Therefore, it is possible to avoid stereotyping of the escape route of the escapee.

[0147] In addition, the operator can participate in the activities of the complex amusement facility 10 from a place far away from the complex amusement facility 10. And, since the operator can enjoy the discount service when visiting the complex amusement facility 10, this service becomes an incentive for the operator to escape, and a sufficient number of escapees can join the pursuit game.

[0148] Furthermore, since the provider of the display system does not need to create an algorithm for setting the escape route of the escapee, the development cost can be reduced accordingly.

[0149] <Another Example of Display System>

[0150] exist Figure 15 to Figure 17Another example of the display system of this embodiment is shown in FIG. In this system, the server 70 creates an overlapping image in which the image of the escapee, i.e., the imaginary object, is overlapped with the captured image of the real world. In addition, the server 70 creates a game image in which the images of the escapee and the pursuer are overlapped with the park map data.

[0151] Reference Fig.15 , on server 70, except for Figure 4 In addition to the functional blocks shown in FIG. 8 , a display control unit 88 is also provided. Figure 4 The repeated functional blocks have the same functions, so the description thereof will be omitted as appropriate. The display control unit 88 includes a display device 30 (see Figure 6 ) and the display control unit 45 and the remote terminal device 100 (refer to Figure 7 ) The display control unit 105 in FIG. 106 is a function of the display control unit 105 in FIG. 107 . That is, the display control unit 88 generates an augmented reality image displayed on the display device 30 and a game image displayed on the remote terminal device 100 .

[0152] In addition, the server 70 is provided with a display control unit 88. Fig.16 As shown in the example, the display control unit 45 is omitted in the display device 30. In addition, the virtual object data storage unit 57 is also omitted. Fig.17 As illustrated, in the remote terminal device 100 , the virtual object data storage unit 112 and the facility map storage unit 113 are omitted except for the display control unit 105 .

[0153] exist Fig.18 The example is based on Fig.15 , Fig.16 The tracking process of the tracking game on the server 70 and the display device 30. Figure 8 The processing contents of the steps with the same figure marks in the tracker process do not change, so the description is appropriately omitted below.

[0154] When receiving participation registration for the chase game from the display device 30, the display device 30 transmits participation registration information to the server 70. This information also includes position information of the display device 30. The participation registration information is stored in the chase game participant storage unit 80 of the server 70.

[0155] Furthermore, in steps S16 to S22, the display device 30 recognizes the real objects in the captured image and obtains the position information of these real objects in the world coordinate system. Next, the captured image data with the position information is sent to the server 70 together with the position information of the display device 30 (S82). Furthermore, the display control unit 88 of the server 70 obtains the position coordinates of the escapee included in the captured image (S84).

[0156] In the next step S28, when the runner is not captured by the pursuer (i.e., the holder of the display device 30), i.e., the tracking game is in progress, the display control unit 88 causes the character image of the runner, i.e., the imaginary object, to overlap the position of the runner in the captured image (S86). The overlapped character image is determined based on the position of the runner. For example, the character image set corresponding to the area 14A, 16A, 18A containing the position of the runner is overlapped with the captured image. Furthermore, the overlapped image (AR image) is sent from the server 70 to the display device 30 (S88).

[0157] exist Fig.19 The example is based on Fig.15 , Fig.17 The process of tracking the runner of the game by the server 70 and the remote terminal device 100. Fig.13 The steps with the same figure marks in the escape process do not change their processing content, so their descriptions are appropriately omitted below.

[0158] In step S58, when the participant calculation unit 84 of the server 70 obtains the position information of the pursuer and the escapee from the pursuit game participant storage unit 80, the display control unit 88 creates a game image in which the character images of the pursuer and the escapee are added to the park map image (S90). In addition, an identification image such as a shadow is added to the escapee corresponding to the remote terminal device 100 to which the game image is sent. The created game image is sent to the remote terminal device 100 (S92) and displayed on the display unit 106 of the remote terminal device 100.

[0159] According to the above-described embodiment, since the display control unit is omitted from the display device 30 and the remote terminal device 100 , the load on these devices to create augmented reality images and game images is reduced.

[0160] <Another Example of Display Device>

[0161] In the above-mentioned embodiment, a smartphone as a video see-through display is exemplified as the display device 30, but the display device 30 of this embodiment is not limited to this embodiment. Fig. 20 Like the head mounted display (HMD) as exemplified above, the display device 30 may be constituted by an optical see-through display.

[0162] In this case, the display device 30 includes the camera 35 , a half mirror 202 corresponding to the display unit 46 , a projector 203 corresponding to the display control unit 45 and the image recognition unit 58 , and a sensor unit 201 corresponding to the position information acquisition unit 50 .

[0163] The half mirror 202 may be, for example, a lens of glasses or goggles. The half mirror 202 transmits light (image) from the real world to the wearer. On the other hand, the projector 203 disposed above the half mirror 202 projects an image of a virtual object onto the half mirror 202. Thus, an augmented reality image can be displayed in which the image of the virtual object, i.e., the character, representing the escapee is superimposed on the scene in the real world, i.e., the scene in the complex amusement facility 10.

Claims

1. A display system comprising a display device and an operating device, The display device includes a camera, an image recognition unit, and a display control unit. The camera shoots scenes of the real world. The image recognition unit recognizes real objects in the image captured by the camera, and recognizes the position of each recognized real object in space coordinates. The display control unit causes a virtual object to be displayed in an overlapping manner at a designated position on the spatial coordinates in the captured image. The image recognition unit recognizes a posture of the real object at the designated position of the virtual object in the captured image, The display control unit performs image processing on the virtual object according to the recognized posture, The operating device includes an input unit capable of inputting an operating instruction for changing the designated position of the virtual object. The display system further includes a server capable of communicating with the display device and the operating device. The display device is arranged in a complex facility, wherein the complex facility is provided with a plurality of facilities having respective different specific themes. The server includes a storage unit and a sending unit. The storage unit stores the character images set as the characters in each of the facilities in association with the position information of the complex facility. The transmitting unit transmits the data of the character image together with the position information associated with the character image to the display device, The display control unit selects the character image corresponding to the designated position as the image of the virtual object and displays the image superimposed on the captured image.

2. The display system according to claim 1, The posture is a capture action at the specified position, The display device sends a capture notification to the server when the image recognition unit recognizes the capture action of the real object. The server includes a reward calculation unit that determines a reward for an operator of the operating device based on an escape period from a time point when the operation of the virtual object is started to a time point when the capturing action is recognized.

3. The display system according to claim 2, The display device includes an input unit for inputting registration of participation as a tracker of a tracking game for tracking the virtual object. The operation device includes an input unit for inputting registration of participation in the pursuit game by an operator of the virtual object. The server includes a participant calculation unit, which calculates the number of the trackers and the operators who are participating in the tracking game, and obtains the positions of the virtual object operated by the operator and the display device held by the tracker in the complex facility. The reward calculation unit determines a reward for the operator based on a spatial density of a virtual object around the captured virtual object at a recognition time point of the capturing action.

4. The display system according to claim 3, The reward calculation unit determines a reward for the operator of the operating device based on a spatial density of trackers around the captured virtual object at a recognition time point of the capturing action.

Images