Display system, display device and program

Abstract

A display system, a display device and a program. The display device includes an image recognition unit, a display control unit, a display unit and a position information acquisition unit. The image recognition unit recognizes a person image from an image captured remotely. The display control unit can overlap a decorative image on the person image, and can generate an augmented reality image in which the decorated person image is overlapped on a scene in the real world. The position information acquisition unit obtains its own position. The server includes a decorative image storage unit and a decorative image extraction unit. The decorative image storage unit stores a plurality of decorative images. The decorative image extraction unit extracts a decorative image based on a position from the decorative image storage unit as a decorative image to be overlapped on the person image.

Description

Technical Field

[0001] This specification discloses a display system, a display device, and a program for displaying an augmented reality (AR) image. Background Art

[0002] Display devices using augmented reality technology are known in the past. For example, Japanese Patent Application Publication No. 2016-522485 displays an augmented reality image in which real objects (real objects) such as toy action figures are replaced with virtual objects (virtual objects) such as animated virtual action figures. Summary of the invention

[0003] Using the aforementioned augmented reality technology, remote communication is possible. For example, in a theme park or other facility, augmented reality technology can be used to overlay images of people in a remote location on the captured image of the facility, creating a scene where people in a remote location are traveling around the facility.

[0004] This specification discloses a display system, a display device, and a program that can display an augmented reality image that follows the surrounding worldview.

[0005] In this specification, a display system is disclosed. The display system includes a display device and a server. The display device includes an image recognition unit, a display control unit, a display unit, and a position information acquisition unit. The image recognition unit recognizes a person image from an image captured remotely. The display control unit can overlap a decorative image on a person image, and can generate an augmented reality image in which the decorated person image is overlapped on a scene in the real world. The display unit can display the augmented reality image. The position information acquisition unit acquires its own position. The server includes a storage unit and an extraction unit. The storage unit stores a plurality of decorative images. The extraction unit extracts a decorative image based on a position acquired by the position information acquisition unit from the storage unit as a decorative image to be overlapped on the person image.

[0006] According to the above configuration, a virtual image obtained by decorating a remote location character image based on the position of the display device can be superimposed on the display unit. This enables a scene presentation in which a remote location character image is decorated in accordance with the world view of the facility where the display device is located.

[0007] In the above configuration, the display device may be disposed in a facility configured based on a specific theme. In this case, the storage unit stores a character image set as a character in the facility as a decorative image.

[0008] According to the above configuration, it is possible to perform a scene presentation of a character in a remote character role-playing facility.

[0009] In the above configuration, a device in the facility may display a photographable identifier. In this case, the display device includes a camera capable of photographing the interior of the facility. In addition, when the image recognition unit recognizes the identifier in the image captured by the camera inside the facility, the display control unit sets the image area containing the identifier as the overlapping area of ​​the decorated person image.

[0010] According to the above configuration, since the overlapping area of ​​the image of a person at a remote location in the captured image within the facility is set, unnatural overlapping such as overlapping of the image of a person in the air is suppressed.

[0011] In the above configuration, the display system may further include a first intercom device capable of communicating within the facility and a second intercom device capable of communicating with the first intercom device remotely.

[0012] According to the above configuration, it is possible to have a conversation with a person superimposed on the augmented reality image within the facility.

[0013] In addition, a display device is disclosed in this specification. The display device includes an image recognition unit, a display control unit, a display unit, a position information acquisition unit, a storage unit, and an extraction unit. The image recognition unit recognizes a person image from an image captured remotely. The display control unit can overlap a decorative image on a person image, and can generate an augmented reality image in which the decorated person image is overlapped on a scene in the real world. The display unit can display an augmented reality image. The position information acquisition unit acquires its own position. A plurality of decorative images are stored in the storage unit. The extraction unit extracts a decorative image based on the position acquired by the position information acquisition unit from the storage unit as a decorative image overlapped on the person image.

[0014] In addition, in this specification, a program is disclosed. The program enables a computer to function as an image recognition unit, a display control unit, a display unit, a position information acquisition unit, a storage unit, and an extraction unit. The image recognition unit recognizes a person image from an image captured remotely. The display control unit can overlap a decorative image on a person image, and can generate an augmented reality image in which the decorated person image is overlapped on a scene in the real world. The display unit can display an augmented reality image. The position information acquisition unit acquires its own position. A plurality of decorative images are stored in the storage unit. The extraction unit extracts a decorative image based on the position acquired by the position information acquisition unit from the storage unit as a decorative image overlapped on the person image.

[0015] According to the display system, display device, and program disclosed in this specification, it is possible to display an augmented reality image that follows the surrounding worldview. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Hereinafter, features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which like reference numerals denote like elements, and wherein:

[0017] Figure 1 This is a diagram illustrating an integrated entertainment facility using the display system of this embodiment and a house serving as a remote location thereof.

[0018] Figure 2 This is a diagram illustrating the interior of a vehicle that can travel in a comprehensive entertainment facility.

[0019] Figure 3 This is a diagram illustrating the hardware configuration of each device included in the display system of this embodiment.

[0020] Figure 4 This is a diagram illustrating functional blocks of the display system according to this embodiment.

[0021] Figure 5 This is a diagram illustrating a display control flow of the display system according to the present embodiment.

[0022] Figure 6 The figure shows an example in which a remote person image is superimposed on a real-world image.

[0023] Figure 7 This is a diagram showing an example in which a remote person image subjected to decoration processing is superimposed on a real-world image.

[0024] Figure 8 This is a diagram illustrating a head mounted display as another example of a display device.

[0025] Fig. 9 This is a diagram showing an example in which a server function is included as another example of the functional block of the display device. DETAILED DESCRIPTION

[0026] Figure 1 1 and 2 , there are illustrated an example of a comprehensive amusement facility 10 using the display system of this embodiment and a house 92 installed at a remote location relative to the facility.

[0027] <Composition of comprehensive entertainment facilities>

[0028] A plurality of theme parks 14 to 18 are provided in the comprehensive entertainment facility 10. The so-called theme park refers to a facility that forms a world view based on a specific theme, and based on the world view, comprehensively forms facilities, events, landscapes, etc. and presents a scene. For example, the theme parks 14 to 18 are connected by a communication path 20A, and users can travel between the theme parks 14 to 18 via the communication path 20A.

[0029] The comprehensive amusement facility 10 includes theme parks with different themes. For example, the comprehensive amusement facility 10 includes an amusement park 14, an aquarium 16, and a zoo 18 as theme parks.

[0030] Each theme park 14 to 18 sets characters based on its own theme. The characters are set according to the theme and world view of each theme park 14 to 18. For example, in the amusement park 14, characters such as clowns and go-karts are set. For example, in the aquarium 16, characters such as dolphins, goldfish, and sharks are set. And, in the zoo 18, characters such as elephants, lions, and pandas are set.

[0031] As described later, these character images (hereinafter referred to as character images as appropriate) are used as decorative images for decorating the human images photographed remotely. The character image data of the theme parks 14 to 18 are stored in the server 70 (see Figure 4 ) of the decorative image storage unit 82. The server 70 is installed in, for example, a building (not shown) for administrators installed in the comprehensive entertainment facility 10. Details of the character image data stored in the decorative image storage unit 82 will be described later.

[0032] In-park passages 20B to 20D are provided in each of the theme parks 14 to 18. The in-park passages 20B to 20D are connected to the communication passage 20A, and the vehicle 90 can travel on these passages.

[0033] exist Figure 2 , the interior of the vehicle 90 is illustrated. The vehicle 90 may be, for example, a small so-called low-speed mobile vehicle that can accommodate four or six people, for example, as a family unit. A camera 35C is provided inside the vehicle 90. The camera 35C can capture the scene inside the comprehensive amusement facility 10, and can display the captured image on a display device 30B (see FIG. 1 ) placed remotely as described later. Figure 1 ). The details of the camera 35C will be described later.

[0034] Beacon transmitters 22 are provided along the communication passage 20A and the park passages 20B to 20D of the comprehensive amusement facility 10. For example, a plurality of transmitters 22 are provided at equal intervals. As described later, the beacon receiver 37 (see FIG. 2 ) of the display device 30A is used to transmit the beacon to the display device 30A. Figure 3 ) receives the signal from the transmitter 22 and can obtain the current position of the display device 30A.

[0035] The display device 30B and the camera 35B are arranged in a house 92 at a remote location away from the comprehensive entertainment facility 10. The display device 30B and the camera 35B are operated by, for example, a person 94 in the house 92. In addition, the camera 35B and the display device 30B can be operated by the person 94 at the remote location and can communicate with the display device 30A and the camera 35C. For example, they can be arranged outside the house instead of being arranged in the house 92.

[0036] As described in detail below, the display system of this embodiment displays an image of a person 94 at a remote location on a display device 30A held by a person attending the comprehensive amusement facility 10. For example, the person attending the comprehensive amusement facility 10 includes a family, and the person 94 at a remote location includes a guardian of the family who is on duty at the remote location, an elderly close relative who has difficulty moving, and the like.

[0037] Furthermore, the display system of this embodiment performs image processing to superimpose a decorative image on the image of the person 94 at the remote location when displaying the image on the display device 30A. As the decorative image, character images set for the theme parks 14 to 18 where the display device 30A is located are extracted.

[0038] <Display system configuration>

[0039] exist Figure 3 2 , the hardware configuration of the devices included in the display system of the present embodiment is exemplified. The display system of the present embodiment is configured to include display devices 30A and 30B, cameras 35B and 35C, and a server 70 .

[0040] Furthermore, if attention is paid to the function of decorating the image of the person 94 at the remote location with a decorative image and superimposing the decorated image on the scene of the comprehensive entertainment facility 10 which is the scene of the real world, the display system only needs to include at least the display device 30A, the camera 35B and the server 70.

[0041] <Configuration of Display Device 30B>

[0042] The display device 30B is provided by a remote person 94 (refer to Figure 1 ) operation and utilization. The display device 30B may be, for example, a smart phone. The display device 30B includes a CPU 31B of a computing device, a system memory 40B of a storage device, and a storage device 41B.

[0043] In addition, the display device 30B includes a display unit 46B and a display control unit 45B that controls the display image of the display unit 46B. In addition, the display device 30B includes an input unit 47B for inputting information. For example, a touch panel display in which the input unit 47B and the display unit 46B are integrated is provided on the display device 30B. In addition, the display device 30B is provided with a communication device 32B (second communication device) that can remotely communicate with the display device 30A. In addition, the display device 30B includes an input / output controller 39B that manages the input / output of information.

[0044] The display device 30B can wirelessly communicate with the display device 30A via a communication line (not shown) and a base station of a communication carrier, etc. The communicated data includes, for example, voice data uttered from a person 94 at a remote location to the intercom 32B.

[0045] The camera 35B is disposed near the display device 30B and can be operated by a remote person 94. The camera 35B is configured to include a camera such as a CMOS or a CCD, and can capture still images and moving images.

[0046] The camera 35B can capture a scene of the real world including the remote person 94. For example, in order to include a full-body image of the remote person 94 in the field of view, the camera 35B is spaced a predetermined distance away from the person 94.

[0047] Furthermore, the camera 35B may be a so-called RGB-D camera that has a function of measuring the distance between the subject and the camera 35B in addition to the function of photographing the real world. As a function of measuring the distance, for example, the camera 35B is provided with a distance measuring mechanism using infrared rays in addition to the above-mentioned photographing device.

[0048] The camera 35B is connected to the communication line and transmits data including an image of a person 94 at a remote location (hereinafter referred to as a remotely photographed image) to the display device 30A. The remotely photographed person image 100 (see FIG. 1 ) can be displayed on the display unit 46A of the display device 30A by receiving the data. Figure 6 ).

[0049] <Configuration of Camera 35C>

[0050] The camera 35C is disposed in the comprehensive amusement facility 10. The camera 35C is configured to include a camera device such as a CMOS and a CCD, similarly to the camera 35B, and can capture still images and moving images. The camera 35C is provided with a distance measuring mechanism using infrared rays.

[0051] Figure 2, an example is shown in which a camera 35C is arranged in a vehicle 90 traveling in a comprehensive amusement facility 10. The camera 35C is arranged on a seat 91 in the vehicle, or more specifically, on its seat surface. The camera 35C may also be a portable device carried by a person attending the comprehensive amusement facility 10. In this case, the camera 35C may be a so-called clone robot. In addition, the camera 35C may also be a fixed device fixed to the seat 91.

[0052] The camera 35C can wirelessly communicate with the display device 30B at a remote location via a communication line and a base station of a communication carrier (not shown), and transmits the captured still image data or moving image data to the display device 30B.

[0053] Reference Figure 2 The camera 35C is provided with a rotation axis perpendicular to the configuration surface (seat surface) and can swing its head around the rotation axis. For example, when the input unit 47B (refer to the input unit 47B of the display device 30B at a remote location) is Figure 3 ) receives an operation signal, the camera 35C changes its field of view by moving its head according to the signal. Thus, the person 94 at a remote location can watch the desired scene in the comprehensive entertainment facility 10 via the display device 30B.

[0054] In addition, the camera 35C displays an AR marker 35C1 as an identifier that can be photographed. The AR marker 35C1 is printed on the surface of the camera 35C. The AR marker 35C1 is an identifier when an augmented reality image is displayed, for example, indicating a display area of ​​an overlapped virtual image. For example, when photographing the inside of the vehicle 90, the image area including the AR marker 35C1 is set as the overlap area of ​​the virtual image.

[0055] In this way, by displaying the AR marker 35C1 as an identifier in the integrated amusement facility 10 in a state where a photograph can be taken, it is possible to set a person image 100 (see Figure 6 ) overlapping area. For example, a scene presentation in which a remote person 94 sits on a seat 91 can be performed.

[0056] <Server Configuration>

[0057] Reference Figure 3 The server 70 is composed of a computer, for example, and is installed in the comprehensive entertainment facility 10 (see Figure 1 ) is in a management building. In addition, the server 70 is wirelessly connected to the display device 30A via a communication means such as a wireless LAN.

[0058] The server 70 includes an input unit 71 such as a keyboard and a mouse, a CPU 72 of a computing device, and a display unit 73 such as a display. In addition, the server 70 includes a ROM 74, a RAM 75, and a hard disk drive 76 (HDD) as a storage device. Furthermore, the server 70 includes an input / output controller 77 that manages the input / output of information. These components are connected to an internal bus 78.

[0059] Figure 4 , the functional blocks of the server 70 are illustrated. This functional block diagram is formed by, for example, the CPU 72 executing a program stored in the ROM 74, the HDD 76, or a computer-readable non-transitory storage medium such as a DVD.

[0060] The server 70 includes a facility map storage unit 80 , a decoration image storage unit 82 , and a decoration image extraction unit 85 .

[0061] The facility map storage unit 80 stores map information within the comprehensive amusement facility 10. For example, location information of passages (communication passages 20A and in-park passages 20B to 20D) and facilities within the comprehensive amusement facility 10 is stored. Specifically, the floor plan data of the comprehensive amusement facility 10 is stored in the facility map storage unit 80 in a state of being associated with the location information. The location information includes longitude and latitude information using the GPS function and location information using the beacon function.

[0062] The decoration image storage unit 82 stores decoration image data that are virtual objects in the augmented reality image displayed on the display device 30A. Figure 7 In FIG. 1 , a lion's head image is shown as a decorative image 110. The decorative image 110 is superimposed on the person image 100 transmitted from a remote location.

[0063] The decoration image data stored in the decoration image storage unit 82 may be 3D model data of a decoration object as a virtual object. The 3D model data includes, for example, 3D image data of the decoration object, and the image data includes shape data, texture data, and motion data.

[0064] A plurality of types of decorative image data are stored in the decorative image storage unit 82 for each of the theme parks 14 to 18. For example, for one theme park, 10 or more and 100 types of decorative image data are stored in the decorative image storage unit 82. These decorative image data are assigned identification codes for each of the theme parks 14 to 18. Furthermore, a unique identification code is assigned to each decorative image data.

[0065] The decorative image data is, for example, character images set as characters in the theme parks 14 to 18. For example, the decorative image 110 with the identification code corresponding to the amusement park 14 includes an image of a large ball for stepping on. Also, the decorative image 110 with the identification code corresponding to the aquarium 16 includes an image of a school of fish.

[0066] In addition, Figure 7 In order to make the illustration clear, outline images are shown as the character image 100 and the decorative image 110, but the present invention is not limited to this form, and 3D images of the character image 100 and the decorative image 110 may be displayed.

[0067] The decoration image extraction unit 85 generates a decoration image based on the position information acquisition unit 50 (see FIG. 5 ) of the display device 30A. Figure 4 ) to determine which theme park 14 to 18 in the comprehensive amusement facility 10 the display device 30A is located in. Furthermore, when the display device 30A is located in a certain theme park 14 to 18, the decorative image extraction unit 85 extracts decorative image data set corresponding to the theme park 14 to 18 located therein from the decorative image storage unit 82, and sends it to the display device 30A.

[0068] <Configuration of Display Device 30A>

[0069] Reference Figure 1 The display device 30A is disposed in the comprehensive amusement facility 10 and is used by the visitors (users) of the facility. The display device 30A can display a virtual reality image obtained by superimposing an image of a virtual object on a scene in the real world.

[0070] The display device 30A may be a portable device, for example, a smartphone including a camera and a display unit, or a glasses-type head mounted display (HMD).

[0071] From the perspective of how the real-world scene is displayed, the display device 30A can be divided into a video see-through display (VST display) and an optical see-through display (OST display). The VST display uses a camera or other image capturing device to capture the real-world scene and displays the captured image on a display unit. On the other hand, the OST display visually recognizes the real-world scene through a see-through display unit such as a half mirror and projects a virtual object on the display unit.

[0072] The above-mentioned smart phone etc. is equipped with a camera 35A (see Figure 3) is classified as a VST display. In addition, the above-mentioned head mounted display (HMD) uses the lens of glasses as a display unit to visually confirm the scene of the real world, so it is classified as an OST display.

[0073] In the following implementation, Figure 6 As an example of the display device 30A, a portable VST display type smartphone is shown in the figure. The smartphone may be owned by the user of the comprehensive amusement facility 10 or may be a rental item such as a tablet terminal lent to the user of the comprehensive amusement facility 10.

[0074] Figure 3 , the hardware configuration of the display device 30A is illustrated. The display device 30A includes a CPU 31A (central processing unit), a talker 32A (first talker), a camera 35A, a GPS receiver 36, a beacon receiver 37, an input / output controller 39A, a system memory 40A, a storage device 41A, a GPU 42, a frame memory 43, a RAMDAC 44, a display control unit 45A, a display unit 46A, and an input unit 47A.

[0075] The system memory 40A is a storage device used by an operating system (OS) executed by the CPU 31A. The storage device 41A is an external storage device and stores, for example, a program for displaying a virtual reality image (AR image) described later.

[0076] The camera 35A is, for example, a camera device mounted on a smartphone, and can capture scenes of the real world in still images and moving images. The camera 35A is, for example, configured to include a camera device such as a CMOS and a CCD. In addition, the camera 35A may be a so-called RGB-D camera that has a function of measuring the distance from the camera 35A in addition to capturing the real world. As a function of measuring the distance, for example, in the camera 35A, in addition to the above-mentioned camera device, a distance measuring mechanism using infrared rays is also provided.

[0077] The GPU 42 (Graphics Processing Unit) is a computing device for image processing, and mainly works when performing image recognition described later. The frame memory 43 is a storage device for storing images captured by the camera 35A and then processed by the GPU 42. The RAMDAC 44 (Random Access Memory Digital-to-Analog Converter) converts the image data stored in the frame memory 43 into an analog signal for the display unit 46A as an analog display.

[0078] The GPS receiver 36 receives data from the GPS satellites 24 (see Figure 1 ) receives a GPS signal as a positioning signal. The GPS signal contains the position coordinate information of latitude, longitude and altitude. The beacon receiver 37 receives the position signal output from the transmitter 22 of the beacon installed in the comprehensive entertainment facility 10 represented by the communication passage 20A and the park passages 20B to 20D.

[0079] Here, the GPS receiver 36 and the beacon receiver 37 have overlapping position estimation functions. Therefore, the display device 30A may be provided with only one of the GPS receiver 36 and the beacon receiver 37 .

[0080] The input unit 47A can input an activation command or a shooting command to the imager 35A. For example, the input unit 47A may be a touch panel integrated with the display unit 46A.

[0081] The display control unit 45A can generate an augmented reality image (AR image) in which an image of a virtual object is superimposed on a scene in the real world and display it on the display unit 46A. For example, the display control unit 45A superimposes a virtual image on an AR marker 35C1 (see FIG. 1 ) in an image of the interior of the comprehensive amusement facility 10. Figure 2 ) image area. The virtual image includes a decoration image 110 (refer to Figure 7 ) of the person image 100. The decoration process is performed by the display control unit 45A. The augmented reality image obtained by superimposing the decorated person image 100 on the image in the facility is displayed on the display unit 46A by the display control unit 45A. The display unit 46A may be, for example, a liquid crystal display or an organic EL display.

[0082] Figure 4 , a functional block diagram of the display device 30A is illustrated. The functional block diagram is formed, for example, by the CPU 31A and the GPU 42 executing a program stored in the system memory 40A and the storage device 41A. Alternatively, the functional block diagram is formed by the CPU 31A and the GPU 42 executing a program stored in a non-transient storage medium readable by a computer, such as a DVD or a hard disk of a computer. Figure 4 Functional blocks illustrated.

[0083] Figure 4 In Figure 3 1 and 2 show the configuration of the display device 30A in a state where a part of the hardware configuration is illustrated and the functional blocks are mixed. Figure 4 In FIG. 1 , as the hardware configuration, a camera 35A, a display control unit 45A, and a display unit 46A are exemplified.

[0084] The display device 30A includes a position information acquisition unit 50 and an image recognition unit 58 as functional blocks. The display device 30A includes a learned model storage unit 59 as a storage unit. These functional blocks are composed of the CPU 31A, the system memory 40A, the storage device 41A, the GPU 42, and the frame memory 43.

[0085] The position information acquisition unit 50 obtains Figure 3 The device obtains information about its current position from at least one of the GPS receiver 36 and the beacon receiver 37. The position information is the position information of the so-called world coordinate system, and in the case of the GPS signal, the latitude, longitude and altitude information are included in the position information. In addition, in the case where the received position information is obtained from the beacon signal, for example, the x-coordinate and y-coordinate of the plane coordinate system with an arbitrary point of the comprehensive entertainment facility 10 as the origin are included in the position information.

[0086] The image recognition unit 58 receives the captured image data obtained by the camera 35A, 35B and performs image recognition. Image recognition includes recognition of an object in the captured image and estimation of the distance between the object and the display device 30A, the camera 35B. In such image recognition, the captured image data includes, for example, the distance data of each object in the color image data from the camera 35A, 35B in addition to the color image data obtained by capturing the scene in the real world as described above.

[0087] The image recognition unit 58 uses the learning model for image recognition stored in the learned model storage unit 59 to recognize the captured image. In the learned model storage unit 59, for example, a neural network for image recognition that has been learned through an external server is stored. For example, segmented and annotated data of each object in the image, including image data of the outdoor of the comprehensive entertainment facility 10, is prepared as training data. In addition, training data for recognizing human images from captured images is also prepared. Using these training data, a multi-layer neural network is formed after machine learning through teacher learning, and is stored in the learned model storage unit 59. The neural network can be, for example, a convolutional neural network (CNN).

[0088] As described later, by using the learned model, the image recognition unit 58 can recognize and extract a person image from an image captured by the remote camera 35B. In addition, by using the learned model, the image recognition unit 58 can recognize the AR marker 35C1 as an identifier displayed in the comprehensive entertainment facility 10.

[0089] <Augmented reality image display process>

[0090] Figure 5, the display process of the augmented reality image of the display system of this embodiment is illustrated. Hereinafter, the augmented reality image includes an image of a person in a remote location away from the facility superimposed on an image of a scene in the comprehensive entertainment facility 10. In addition, the augmented reality image includes an image of a decorated person image obtained by superimposing a virtual decorative image on the person image superimposed on an image of a scene in the comprehensive entertainment facility 10.

[0091] Figure 5 The display flow shown in the example is, for example, executed by the CPU 31A, 72 and stored in the display device 30A (see Figure 3 ) system memory 40A and storage device 41A, and the ROM 74 and hard disk drive 76 of the server 70. Alternatively, the CPU 31A and 72 execute a program stored in a non-transient storage medium such as a DVD or a computer hard disk that can be read by a computer. Figure 5 Display process.

[0092] In addition, Figure 5 In FIG. 1 , the step executed by the display device 30A is shown as (D), and the step executed by the server 70 is shown as (S). In addition, as a preparatory step for executing the process, at a remote location, the character 94 (refer to Figure 1 ) is photographed by the camera 35B. In addition, in the comprehensive entertainment facility 10, for example, in the vehicle 90 ( Figure 2 ) is provided with a camera 35C on a seat 91 of the integrated amusement facility 10, and the image, that is, the image captured in the facility, is transmitted to the display device 30B at a remote location. Furthermore, the intercom 32A (first intercom) of the display device 30A in the integrated amusement facility 10 and the intercom 32B (second intercom) of the display device 30B at a remote location are in a state where communication is possible.

[0093] Reference Figure 5 and Figure 3 , Figure 4 This process starts when a photographing command is input from the input unit 47A of the display device 30A. The photographing device 35A photographs an image of the real world, that is, an image inside the comprehensive amusement facility 10, based on the photographing command. The obtained data of the photographed image inside the facility is sent to the image recognition unit 58.

[0094] The image recognition unit 58 performs image recognition on the received image captured in the facility (S10). The image recognition includes recognition of the AR marker 35C1 as an identifier contained in the image captured in the facility. In addition, the recognition includes segmentation and annotation of objects in the image captured in the facility. In addition, in the image recognition, the distance of each object from the display device 30A is calculated.

[0095] The image recognition unit 58 determines whether the AR marker 35C1 is recognized in the captured image (S12). If the AR marker 35C1 is not recognized, the process ends. On the other hand, if the AR marker 35C1 is recognized in the captured image, the image recognition unit 58 tracks the AR marker 35C1 for a predetermined period (so-called marker tracking) and determines whether the AR marker 35C1 is included in the captured image for the predetermined period (S14). The predetermined period may be, for example, not less than 5 seconds and not more than 10 seconds.

[0096] If the AR marker 35C1 disappears from the captured image during the predetermined period, it can be considered as a so-called inadvertent projection, and the generation of the augmented reality image initiated by the AR marker 35C1 is skipped. In other words, the display of the augmented reality image on the display unit 46A is retained. On the other hand, if the AR marker 35C1 is included in the captured image for a predetermined period, the image recognition unit 58 sets the image area of ​​the AR marker 35C1 as the person image 100 (see Figure 7 ) and the overlapping area of ​​the decorative image 110.

[0097] Furthermore, the current position of the display device 30A is obtained by the position information acquisition unit 50. The current position information is sent to the server 70 (S16). When the server 70 receives the current position information of the display device 30A, the decorative image extraction unit 85 confirms which position of the comprehensive amusement facility 10 the current position of the display device 30A corresponds to based on the park map data stored in the facility map storage unit 80. Furthermore, the decorative image extraction unit 85 determines whether the current position of the display device 30A is included in any of the theme parks 14 to 18 (S18).

[0098] When the current position of the display device 30A is out of any of the theme parks 14 to 18, the decorative image extraction unit 85 notifies the display device 30A that the current position is out of the theme parks 14 to 18 (S20). Figure 1 ) is traveling on the communication path 20A, such an out-of-park notification is sent to the display device 30A.

[0099] Receiving the notification of being outside the park, the image recognition unit 58 of the display device 30A acquires data of a photographed image of the remote location from the camera 35B of the remote location. Furthermore, the image recognition unit 58 recognizes a person image from the acquired photographed image of the remote location (S22).

[0100] The image recognition unit 58 extracts the person image 100 (see Figure 6), and sends the image data to the display control unit 45A. In addition, the image recognition unit 58 sends the data of the captured image inside the facility in which the image recognition has been performed, more specifically, the AR marker 35C1 has been recognized, to the display control unit 45A.

[0101] The display control unit 45A is as follows Figure 6 As shown in the example, the augmented reality image obtained by superimposing the person image 100 on the image 105 taken inside the facility is displayed on the display unit 46A (S24). For example, the display control unit 45A obtains the center of gravity of the image area of ​​the AR marker 35C1 and the center of gravity of the person image area. Then, the person image 100 is superimposed on the image 105 taken inside the facility in such a way that the center of gravity of the image area of ​​the AR marker 35C1 coincides with the center of gravity of the person image area. Through such image processing, it is possible to present a scene in which the person 94 in a remote location appears to be in the comprehensive entertainment facility 10.

[0102] Returning to step S18, when the decoration image extraction unit 85 of the server 70 determines that the current position of the display device 30A is included in any one of the theme parks 14 to 18, the decoration image extraction unit 85 extracts the decoration image 110 (see Figure 7 That is, the decorative image extraction unit 85 extracts the data of the decorative image 110 associated with the theme parks 14 to 18 where the display device 30A stays from the decorative image storage unit 82 ( S26 ).

[0103] For example, the decorative image extraction unit 85 extracts data of the decorative image 110 with identification codes of the theme parks 14 to 18 stored in the display device 30A. When a plurality of decorative images are stored for each of the theme parks 14 to 18, an arbitrary image is extracted from the decorative images with identification codes of the same theme park.

[0104] Then, the decorative image extraction unit 85 sends the extracted decorative image to the display device 30A (S28). Then, the image recognition unit 58 of the display device 30A recognizes and extracts a person image from the remote location captured image obtained from the remote location camera 35B (S30). The display control unit 45A of the display device 30A performs a decorative process of superimposing the decorative image 110 obtained from the decorative image extraction unit 85 on the remote location person image 100 obtained from the image recognition unit 58 (S32).

[0105] For example, during the decoration process, the display control unit 45A establishes a correspondence between the body parts of the person image 100 and the body parts of the decorative image. For example, when the decorative image 110 is an image of an animal's head, the image recognition unit 58 performs image recognition on the person image 100 to estimate the head of the person image 100. Furthermore, the display control unit 45A sets the head area of ​​the person image 100 as the overlapping area of ​​the decorative image 110.

[0106] Furthermore, the display control unit 45A is as follows Figure 7 As illustrated, an augmented reality image obtained by overlapping the decorated (decorated) person image 100 with the image 105 taken inside the facility is displayed on the display unit 46A (S34). For example, the display control unit 45A finds the center of gravity of the image area of ​​the AR marker 35C1, and finds the center of gravity of the decorated person image area. Furthermore, the person image 100 is overlapped with the image 105 taken inside the facility in such a way that the center of gravity of the image area of ​​the AR marker 35C1 coincides with the center of gravity of the person image area. Through such image processing, a scene presentation can be performed in which a person 94 in a remote location plays the role of the theme park 14 to 18 that the attendees of the comprehensive entertainment facility 10 have come to.

[0107] <Another Example of Display Device>

[0108] In the above-mentioned embodiment, a smartphone as a video see-through display is exemplified as the display device 30A, but the display device 30A of the present embodiment is not limited to this form. Figure 8 Like the head mounted display (HMD) shown as an example, the display device 30A is constituted by an optical see-through display.

[0109] In this case, the display device 30A includes a camera 35A, a half mirror 114 corresponding to the display unit 46A, a projector 116 corresponding to the display control unit 45A and the image recognition unit 58 , and a sensor unit 112 corresponding to the position information acquisition unit 50 .

[0110] The half mirror 114 may be, for example, a lens of glasses or goggles. The half mirror 114 transmits light (image) from the real world to the user. On the other hand, the projector 116 disposed above the half mirror 114 projects an image of a virtual object onto the half mirror 114. Thus, an augmented reality image can be displayed in which a remote person image 100 or a person image 100 superimposed with a decorative image 110 is superimposed on a scene in the comprehensive entertainment facility 10 as a scene in the real world.

[0111] <Another Example of Display Device>

[0112] In the above-mentioned embodiment, the display device 30A and the server 70 perform Figure 5 However, the display device 30A may perform all the steps of the process instead of the display device 30A. In this case, the display device 30A is composed of a tablet terminal with a larger storage capacity than a smartphone, for example.

[0113] Fig. 9 In the example, Figure 4 The functional block diagram of the display device 30A is a modified example of the present invention. The functional block diagram is formed by, for example, the CPU 31A and the GPU 42 executing a program stored in the system memory 40A and the storage device 41A. Alternatively, the CPU 31A and the GPU 42 execute a program stored in a non-transient storage medium readable by a computer, such as a DVD or a hard disk of a computer. Fig. 9 Function block.

[0114] exist Fig. 9 In, with Figure 4 In contrast, the display device 30A is provided with a facility map storage unit 80, a decoration image storage unit 82, and a decoration image extraction unit 85. Figure 4 These components provided on the server 70 are provided on the display device 30A, and the virtual reality image display process can be performed by the display device 30A alone. Figure 5 In the process of , all steps are executed by the display device 30A. Furthermore, the data exchange between the display device 30A and the server 70 is no longer required, so steps S16 and S28 are no longer required.

[0115] <Another example of identifier>

[0116] In the above-mentioned embodiment, as an identifier for the display device 30A to generate an augmented reality image, an AR marker 35C1 is attached to the surface of the camera 35C, but the display system of this embodiment is not limited to this form. Figure 2 ) A markerless AR method in which the AR marker 35C1 is not attached. Specifically, the three-dimensional shape of the camera 35C can be used as an identifier.

Claims

1. A display system, A display device and a server capable of communicating with the display device are provided. The display device comprises: An image recognition unit that recognizes a person image from an image captured at a remote location; a display control unit capable of superimposing a decorative image on the character image, and capable of generating an augmented reality image in which the decorated character image is superimposed on a scene in the real world; A display unit capable of displaying the augmented reality image; as well as The location information acquisition unit acquires the location of the device. The server has: a storage unit storing a plurality of the decorative images corresponding to each area in a facility composed of a plurality of areas; and an extracting unit, which determines the area where the display device is located based on the position of the display device itself, and extracts the decorative image corresponding to the area where the display device is located from the storage unit as the decorative image superimposed on the character image, The display device is arranged in a facility constructed based on a specific theme. In the storage unit, a character image set as a character in the facility is stored as the decorative image, The equipment in the facility displays an identifier that can be photographed, The display device includes a camera capable of photographing the interior of the facility. When the image recognition unit recognizes the identifier in the image captured inside the facility captured by the camera, the image recognition unit tracks the identifier for a predetermined period of not less than 5 seconds and not more than 10 seconds, and determines whether the identifier is continuously included in the image captured inside the facility for the predetermined period. If the identifier disappears from the image captured within the facility during a predetermined period, skipping the generation of the augmented reality image initiated by the identifier, When the identifier is included in the image captured inside the facility for a predetermined period of time, the display control unit sets the image area including the identifier as the overlapping area of ​​the decorated image of the person, and the display control unit calculates the center of gravity of the image area of ​​the identifier and the center of gravity of the area of ​​the decorated image of the person, so as to overlap the decorated image of the person on the image captured inside the facility in a manner that the center of gravity of the image area of ​​the identifier is consistent with the center of gravity of the area of ​​the decorated image of the person.

2. The display system according to claim 1, It has a first intercom and a second intercom, The first intercom device is capable of conducting calls within the facility, The second intercom device can conduct a conversation with the first intercom device at the remote location.

Images