Display system and display device

Abstract

The present disclosure relates to a display system and a display device. When an augmented reality display device uses a theme park, which is a facility based on a specific theme, as a departure point, the augmented reality display device sends the theme park information of the departure point to a server. The server has a theme park-specific character storage unit and a sending unit. The theme park-specific character storage unit stores information about characters set for the theme park of the departure point. The sending unit uses the character set for the theme park of the departure point as a virtual object of the departure point and sends image data of the virtual object to the augmented reality display device.

Description

Technical Field

[0001] This specification discloses a display system capable of displaying an augmented reality image (AR image) and a display device included in the display system. Background Art

[0002] Conventionally, display devices using augmented reality (AR) technology are known. For example, in Japanese Patent Application Laid-Open No. 2014-20863, an augmented reality image (AR image) obtained by superimposing a virtual image of a character traveling along a route to a destination on an image of the real world is displayed on a display in a navigation system. Summary of the invention

[0003] Furthermore, for example, if users of a theme park whose worldview is constructed based on a specific theme can be provided with services based on the worldview even after they leave the theme park, this will contribute to improving customer satisfaction.

[0004] Therefore, this specification discloses a display system that can provide users of a facility such as a theme park with a service based on the world view of the facility even after the users leave the facility, and a display device included in the system.

[0005] In this specification, a display system is disclosed. The display system includes a display device and a server wirelessly connected to the display device. The display device can move together with the user. In addition, the display device includes a display unit and a display control unit. The display control unit can display an augmented reality image obtained by superimposing an image of a virtual object on a real-world scenery on the display unit. When the display device uses a facility constructed based on a specific theme as a departure point, the display device sends facility information of the departure point to the server. The server includes a storage unit and a sending unit. Information on a character set for the facility at the departure point is stored in the storage unit. The sending unit uses the character set for the facility at the departure point as a virtual object at the departure point and sends image data of the virtual object to the display device.

[0006] According to the above configuration, since the characters set for the facility such as the theme park at the departure point are displayed in the virtual reality image, it is possible to provide the user with a service based on the world view of the facility even after the user leaves the facility.

[0007] In addition, in the above-mentioned configuration, the display device can also be mounted on a vehicle. In this case, the display unit is configured in a manner that separates a plurality of seats in the vehicle interior, and the first display surface and the second display surface opposite to the first display surface are respectively oriented to the side of the vehicle interior. In addition, the display device includes a first camera (imager), a second camera, and an image recognition unit. The first camera shoots the second display surface side of the vehicle interior, and the captured image is displayed on the first display surface. The second camera shoots the first display surface side of the vehicle interior, and the captured image is displayed on the second display surface. The image recognition unit performs image recognition of the captured images of the first camera and the second camera. When the image recognition unit recognizes a person in the captured image, the display control unit generates a captured image in which the image area of ​​the recognized person is replaced with a virtual object of the departure point, and displays the captured image on the first display surface and the second display surface.

[0008] According to the above configuration, the characters (passengers) adjacent to each other across the display unit are displayed in place of the virtual objects, so that the user can experience as if they are sitting in the vehicle with the virtual objects.

[0009] In addition, in the above configuration, the display device may also be mounted on a vehicle. In this case, the display unit is mounted on a vehicle window and the display surface faces the vehicle interior. The display device includes a camera and an image recognition unit. The camera captures scenery outside the vehicle. The image recognition unit performs image recognition of an image captured by the camera. When the image recognition unit recognizes a person in the captured image, the display control unit generates a captured image in which the image area of ​​the recognized person is replaced with a virtual object of the departure point, and displays the captured image on the display surface.

[0010] According to the above configuration, since the characters outside the vehicle are replaced with virtual objects and displayed on the display surface serving as the vehicle window, the user can experience the feeling of being in a vehicle moving in a facility such as a theme park.

[0011] In the above configuration, the display control unit may reduce the ratio of the image area of ​​the person replaced by the virtual object of the departure point to the image areas of all recognized persons as the distance from the facility of the departure point increases.

[0012] According to the above configuration, it is possible to realize a performance in which the farther away from the departure facility, the fewer characters of the facility see off the passengers.

[0013] In the above configuration, the storage unit may store information about a character set for a facility as a destination. In this case, the transmission unit uses the character set for the facility as a virtual object of the destination and transmits image data of the virtual object to the display device. The display control unit increases the ratio of the image area of ​​the character replaced with the virtual object of the destination relative to the image areas of all recognized characters as the closer to the facility as the destination.

[0014] According to the above configuration, it is possible to realize a performance in which more characters of a facility greet passengers as the facility is closer to the destination.

[0015] In the above configuration, a plurality of characters may be set in the storage unit as the virtual object of the departure point and the virtual object of the destination, respectively. In this case, the display control unit selects the virtual object character to be replaced according to the time period.

[0016] According to the above configuration, it is possible to realize a performance such as displaying diurnal animals as virtual objects during the day and displaying nocturnal animals as virtual objects after sunset.

[0017] In addition, in this specification, a display device mounted on a vehicle is disclosed. The display device includes a storage unit, an extraction unit, a display unit, and a display control unit. The storage unit stores information about characters set for facilities constructed based on a specific theme. The extraction unit extracts characters set for facilities set as departure points as virtual objects of departure points. The display unit faces the interior of the vehicle with a display surface. The display control unit is capable of displaying an augmented reality image obtained by superimposing an image of a virtual object of departure point on a real-world landscape on the display unit.

[0018] According to the display system and display device disclosed in this specification, even after users of a facility such as a theme park leave the facility, services based on the world view of the facility can be provided to the users. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Hereinafter, features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which like reference numerals represent like elements, and wherein:

[0020] Figure 1 This is a diagram illustrating an integrated entertainment facility including the display system according to the present embodiment.

[0021] Figure 2 It is a perspective view illustrating a vehicle traveling on the connecting passage.

[0022] Figure 3 This is a diagram illustrating the appearance of the interior of a vehicle.

[0023] Figure 4 This is a diagram illustrating the hardware configuration of a display device and a server in the display system according to the present embodiment.

[0024] Figure 5 This is a diagram illustrating the functional blocks (frames) of a server.

[0025] Figure 6 The diagram illustrates the functional blocks of the display device.

[0026] Figure 7 This is a diagram illustrating the replacement ratio of virtual objects in the departure group.

[0027] Figure 8 This is a diagram showing an example in which an augmented reality image is displayed on a display unit in a vehicle interior.

[0028] Fig. 9 This is a diagram illustrating the initial setting process of augmented reality image display.

[0029] Fig.10 It is a diagram illustrating the augmented reality image display process.

[0030] Fig.11 This is a diagram for explaining a process of replacing a character image region with a virtual object image.

[0031] Fig.12 This is a diagram showing an example in which an augmented reality image obtained by superimposing images of virtual objects of a destination group is displayed on a display unit provided as a vehicle window.

[0032] Fig.13 2 is a diagram illustrating the replacement ratio of virtual objects in the departure place group and the destination group.

[0033] Fig.14 This is a diagram illustrating an initial setting flow when images of virtual objects of a departure place group and a destination group are displayed on an augmented reality image.

[0034] Fig.15 This is a diagram for explaining the augmented reality image display process when the images of virtual objects of the departure place group and the destination group are displayed in the augmented reality image.

[0035] Fig.16 FIG. 1 is a diagram showing an example in which vehicles circulate on a connecting passage.

[0036] Fig.17 This is a diagram showing another example of the functional blocks of the display device. DETAILED DESCRIPTION

[0037] <Composition of comprehensive entertainment facilities>

[0038] exist Figure 110 is shown as an example of a comprehensive amusement facility. The display system according to the present embodiment is used in the facility. A plurality of theme parks 12 to 18 are provided in the comprehensive amusement facility 10. A theme park is a facility that forms a world view based on a specific theme, and comprehensively forms and performs facilities, activities, landscapes, etc. based on the world view.

[0039] The comprehensive amusement facility 10 is provided with theme parks with different themes. For example, in the comprehensive amusement facility 10, as theme parks, there are provided an athletic park 12, an amusement park 14, an aquarium 16, and a zoo 18. The theme parks 12 to 18 are provided with virtual object characters based on their respective themes. Figure 8 In this way, the virtual object is used when the augmented reality image (AR image) is displayed on the display unit 41 (the in-vehicle display unit 41A and the window display unit 41B) in the vehicle 90 .

[0040] The virtual object characters are set to characters that match the themes and worldviews of the theme parks 12 to 18. For example, in the sports park 12, multiple characters such as adventurers, commandos, and ninjas are set as virtual objects. For example, in the amusement park 14, multiple characters such as clowns and karts are set as virtual objects. For example, in the aquarium 16, multiple characters such as dolphins, goldfish, and penguins are set as virtual objects. Furthermore, in the zoo 18, multiple characters such as elephants, pandas, and bears are set as virtual objects. The information of these set virtual objects is stored in the theme park-specific character storage unit 82 of the server 70 (see Figure 5 ). The details of the stored virtual object information will be described later.

[0041] <Vehicle Configuration>

[0042] The theme parks 12 to 18 are connected by a connecting passage 20. The vehicle 90 travels on the connecting passage 20. The vehicle 90 may be, for example, a shared (public) shuttle bus connecting the theme parks 12 to 18. The vehicle 90 may be manually driven by a driver, or may have an automatic driving function. Regarding the automatic driving function, for example, based on the standards of the Society of Automotive Engineers (SAE), the vehicle 90 may be capable of operating at level L4 (highly automatic driving) and level L5 (fully automatic driving).

[0043] The vehicle 90 is equipped with the AR display device 30 according to the present embodiment. Figure 2 2 shows a perspective view of a vehicle 90. On both side surfaces 91 of the vehicle 90, an outside camera 33C of the AR display device 30 for photographing scenery outside the vehicle is provided. Figure 39 shows an example of the interior of a vehicle 90. In the interior, a plurality of seats 92 are arranged in the longitudinal direction of the vehicle 90. In addition, in the interior, a passage 93 is provided in the center in the vehicle width direction. The passage 93 is provided along the longitudinal direction of the vehicle. A row of seats 92 is provided along the longitudinal direction of the vehicle across the passage 93.

[0044] The in-vehicle display unit 41A of the AR display device 30 is provided on the passage 93. A plurality of seats 92, specifically, a row of seats arranged on one side and a row of seats arranged on the other side are separated by the in-vehicle display unit 41A. In other words, the space in the vehicle cabin is divided into a right space and a left space by the in-vehicle display unit 41A. The in-vehicle display unit 41A is a plate-shaped display device, which is arranged in an upright state along the front-rear direction of the vehicle. The in-vehicle display unit 41A is, for example, arranged upright from the ground of the passage 93 to the ceiling of the vehicle cabin.

[0045] Both sides of the in-car display unit 41A serve as display surfaces. That is, the in-car display unit 41A has a first display surface 41A1 facing the side of the vehicle compartment (facing the car window) and a second display surface 41A2 facing away from it (facing the opposite direction). As described later, virtual reality images are displayed on these display surfaces. In addition, in the central part of the first display surface 41A1 and the second display surface 41A2 in the front-rear direction of the vehicle, a frame component 94 is arranged along the height direction. The first in-car camera 33A and the second in-car camera 33B of the AR display device 30 are arranged on the frame component 94.

[0046] As indicated by the dotted line, Figure 3 From the perspective of the first in-vehicle camera 33A, the first in-vehicle camera 33A is configured to be hidden behind the frame member 94, and photographs the scenery on the second display surface 41A2 side of the vehicle interior, that is, the right side of the paper. The photographed image is displayed on the first display surface 41A1. Similarly, the second in-vehicle camera 33B photographs the scenery on the first display surface 41A1 side of the vehicle interior, that is, the left side of the paper. The photographed image is displayed on the second display surface 41A2.

[0047] In this way, the passenger on the left side of the paper can see the interior of the vehicle on the right side of the paper from the first display surface 41A1. Similarly, the passenger on the right side of the paper can see the interior of the vehicle on the left side of the paper from the second display surface 41A2. As described later, an augmented reality image is displayed on the first display surface 41A1 and the second display surface 41A2 by replacing the characters in the images captured by the first in-vehicle camera 33A and the second in-vehicle camera 33B with virtual objects.

[0048] The window display unit 41B of the AR display device 30 is provided as a window of the vehicle 90. That is, the window display unit 41B as a display device is provided instead of a glass window as a window in the vehicle 90. The window display unit 41B is arranged so that the display surface faces the interior of the vehicle.

[0049] The vehicle exterior camera 33C (see FIG. 1 ) of the AR display device 30 is displayed on the vehicle window display unit 41B. Figure 2 ) captured. That is, the scenery of the car window is displayed in the car interior via the outside camera 33C and the car window display unit 41B. As described later, an augmented reality image is displayed on the car window display unit 41B by replacing the characters in the image captured by the outside camera 33C with virtual objects.

[0050] <Server Configuration>

[0051] The display system according to the present embodiment is configured to include an AR display device 30 and a server 70. Figure 4 2 shows the hardware configuration of the AR display device 30 and the server 70. The server 70 is composed of, for example, a computer. In addition, the server 70 is installed in, for example, a management building of the comprehensive entertainment facility 10. The server 70 is wirelessly connected to the AR display device 30 via a communication path such as a wireless LAN.

[0052] The server 70 includes an input unit 71 such as a keyboard and a mouse, a CPU (Central Processing Unit) 72 of a computing device, and a display unit 73 such as a display. In addition, the server 70 includes a ROM (Read-Only Memory) 74, a RAM (Random Access Memory) 75, and a hard disk drive (HDD) 76 as a storage device. Furthermore, the server 70 includes an input / output controller 77 for managing input and output of information. These components are connected to an internal bus 78.

[0053] exist Figure 5 2 shows a functional block diagram of the server 70. This functional block diagram is formed by, for example, the CPU 72 executing a program stored in the ROM 74 and / or the HDD 76 or in a non-transitory storage medium such as a DVD.

[0054] The server 70 includes a receiving unit 83 that receives a signal from an external device such as the AR display device 30. The server 70 also includes a route search unit 84, a 3D (three-dimensional) model extraction unit 85, and a transmission unit 86. The server 70 also includes a facility map storage unit 81 and a theme park-specific character storage unit 82 as storage units.

[0055] The facility map storage unit 81 stores map information in the comprehensive amusement facility 10. For example, location information of passages and facilities in the comprehensive amusement facility 10 including the connecting passage 20 is stored.

[0056] The theme park-specific character storage unit 82 stores information about characters set for the theme parks 12 to 18. The character information may be, for example, image data of each character, and the image data may be 3D model data. The 3D model data includes, for example, shape data, texture data, and motion data of the character.

[0057] The receiving unit 83 receives information about the theme park (hereinafter appropriately referred to as the "departure park") that is the departure point of the vehicle 90 from the AR display device 30. The information about the theme park includes, for example, text information about the name of the departure park. In response to this, the path search unit 84 refers to the facility map storage unit 81 and selects a theme park (hereinafter appropriately referred to as the "destination park") that is the destination. For example, the theme park closest to the departure park or the farthest theme park becomes the destination theme park. Furthermore, the path search unit 84 creates a path (guidance path) on the connecting passage 20 from the departure park to the destination park.

[0058] The information of the departure park is sent to the route search unit 84 and also to the 3D model extraction unit 85. Based on the information of the departure park, the 3D model extraction unit 85 refers to the theme park-specific character storage unit 82 to extract the character set for the departure park. Furthermore, the 3D model extraction unit 85 sets the extracted character as a virtual object of the departure group.

[0059] The guidance route obtained by the route search unit 84 and the image data of the virtual objects of the departure point group extracted by the 3D model extraction unit 85 are transmitted to the AR display device 30 via the transmission unit 86 .

[0060] <AR display device configuration>

[0061] Reference Figure 1 The AR display device 30 can display an augmented reality image (AR image) to the user (passenger) of the vehicle 90. An augmented reality image is an image obtained by superimposing an image of a virtual object on a real-world scene. In addition, as described above, the AR display device 30 is mounted on the vehicle 90 and can move with the user (passenger).

[0062] exist Figure 4The hardware structure of the AR display device 30 is illustrated in FIG. For example, the AR display device 30 is a video see-through display, that is, a camera or other camera captures a scene in the real world and displays the captured image on a display unit. Figure 4 In the example of FIG. 1 , the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the out-vehicle camera 33C correspond to the camera. In addition, the in-vehicle display unit 41A and the window display unit 41B correspond to the display unit.

[0063] The AR display device 30 includes a system memory 31, a CPU (central processing unit) 32, a first in-vehicle camera 33A, a second in-vehicle camera 33B, an outside-vehicle camera 33C, a GPS receiver 34, an input / output controller 35, a storage 36, a GPU 37, a frame memory 38, a RAMDAC 39, a display control unit 40, an in-vehicle display unit 41A, and a window display unit 41B.

[0064] The system memory 31 is a storage device used by an operating system (OS) executed by the CPU 32. The storage 36 is an external storage device, and stores, for example, a program for displaying a virtual reality image (AR image) described later.

[0065] The first in-vehicle camera 33A, the second in-vehicle camera 33B, and the out-vehicle camera 33C are configured as imaging devices including, for example, CMOS, CCD, etc. Furthermore, these cameras 33A to 33C may be so-called RGB-D cameras that have a function of measuring the separation distance from each camera 33A to 33C in addition to photographing the real world. As a function of measuring the separation distance, for example, in the cameras 33A to 33C, in addition to the above-mentioned imaging device, a distance measuring mechanism using infrared rays is provided.

[0066] The interior display unit 41A and the window display unit 41B may be, for example, a liquid crystal display device and / or an organic EL display device.

[0067] The GPU (Graphics Processing Unit) 37 is a computing device for image processing, and mainly works when performing image recognition described later. The frame memory 38 is a storage device that stores images captured by the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the outside-vehicle camera 33C and then processed by the GPU 37. The RAMDAC (Random Access Memory Digital-to-Analog Converter) 39 converts the image data stored in the frame memory 38 into analog signals suitable for the in-vehicle display unit 41A and the window display unit 41B.

[0068] The GPS receiver 34 receives data from the GPS satellites 24 (see Figure 1 ) receives a positioning signal, namely a GPS signal. The GPS signal contains position coordinate information of latitude, longitude, and altitude. By obtaining the position coordinate information, the current position of the vehicle 90 can be grasped.

[0069] exist Figure 6 2 shows a functional block diagram of the AR display device 30. The functional block diagram is formed by, for example, the CPU 32 and / or the GPU 37 executing a position estimation program stored in the system memory 31 and / or the storage 36 or in a non-transitory storage medium such as a DVD and / or a computer hard disk.

[0070] exist Figure 6 In Figure 4 Part of the hardware structure and the functional blocks are also shown in a mixed state. Figure 6 In FIG. 1 , as the hardware configuration, a first in-vehicle camera 33A, a second in-vehicle camera 33B, an out-vehicle camera 33C, a display control unit 40 , an in-vehicle display unit 41A, and a window display unit 41B are exemplified.

[0071] In addition, the AR display device 30 includes, as functional blocks, a transmission unit 51, a position estimation unit 52, a navigation function unit 53, a reception unit 54, an image recognition unit 57, a virtual object data storage unit 55, a replacement ratio storage unit 56, and a learned model storage unit 58. These functional blocks are composed of the CPU 32, the system memory 31, the storage 36, the GPU 37, the frame memory 38, and the like.

[0072] The virtual object data storage unit 55 stores the 3D model extraction unit 85 (see Figure 5 ) image data (3D model data) of the virtual objects of the characters in the departure group extracted.

[0073] The replacement ratio storage unit 56 stores Figure 7The replacement ratio map is a graph with the path on the horizontal axis and the replacement ratio on the vertical axis. In this graph, a characteristic line L1 representing the replacement ratio of the virtual objects in the departure group is drawn. In addition, the departure group refers to a plurality of virtual object groups (character groups) set for the departure park.

[0074] The replacement ratio is a value indicating the ratio of the image area of ​​the person replaced by the virtual object to the image areas of all the persons recognized by the image recognition unit 57. The replacement ratio is 100% when the vehicle 90 is located at the departure park, for example. That is, at the departure park, the image areas of all the persons recognized by the image recognition unit 57 are replaced by the virtual objects of the departure group.

[0075] The replacement ratio decreases as the distance from the departure park increases. Furthermore, at the destination park, the image areas of all characters recognized by the image recognition unit 57 are not replaced with virtual objects. In other words, the replacement ratio becomes 0%. By setting such a replacement ratio, it is possible to achieve a performance such as the farther away from the departure park, the fewer characters of the theme park will see off the passengers.

[0076] The learned model storage unit 58 stores, for example, a neural network for image recognition that has been learned by an external server or the like. For example, image data of the interior and exterior of a vehicle, that is, data on which segmentation and labeling of objects, especially people, in the image are completed, are prepared as training data. Using the training data, a multi-layer neural network that is learned by the machine through supervised learning is formed and stored in the learned model storage unit 58. The neural network may be, for example, a convolutional neural network (CNN).

[0077] The image recognition unit 57 receives the captured image data obtained by the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the out-vehicle camera 33C, and performs image recognition on the captured image data. Image recognition includes recognition of an object in the captured image and estimation of the separation distance between the object and the AR display device 30. When performing such image recognition, the captured image data includes, for example, the separation distance data of each object in the color image data from each camera 33A to 33C in addition to the color image data obtained by capturing the scenery of the real world as described above.

[0078] The image recognition unit 57 uses the learned image recognition neural network stored in the learned model storage unit 58 to determine whether the captured images obtained by the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the outside-vehicle camera 33C contain a person. The image area of ​​the person in the captured image recognized by the image recognition unit 57 is distinguished (segmented) from other image areas. Such processed captured images are sent to the display control unit 40.

[0079] The display control unit 40 receives the processed (person recognition has been performed) captured image data from the image recognition unit 57, and also obtains the current position information from the position estimation unit 52. In addition, the display control unit 40 obtains the guidance route information to the destination park from the navigation function unit 53. Furthermore, the display control unit 40 obtains the image data (3D model data) of the virtual object of the departure place group from the virtual object data storage unit 55, and obtains the replacement ratio map from the replacement ratio storage unit 56.

[0080] The display control unit 40 obtains the replacement ratio in the replacement ratio map corresponding to the current position based on the current position provided by the position estimation unit 52 and the guidance route information provided by the navigation function unit 53. Furthermore, the display control unit 40 performs replacement processing of the image area of ​​the character and the virtual object based on the obtained replacement ratio. The replaced image, that is, the augmented reality image, is displayed on the in-vehicle display unit 41A and the window display unit 41B.

[0081] <Augmented reality image display process showing virtual objects of the departure place group>

[0082] exist Figure 8 2 illustrates an example of an augmented reality image display performed by the display system according to the present embodiment. In the figure, a virtual reality image obtained by superimposing a virtual object image 102 on a captured image of the real world is displayed on the first display surface 41A1 and the window display unit 41B. In addition, in order to make the illustration clear, the virtual object image 102 is represented by a plane outline diagram, but it is not limited to this method, and the virtual object image 102 can be a 3D image.

[0083] exist Fig. 9 , a flowchart of the augmented reality image display process performed by the display system according to the present embodiment, especially during initial setting, is illustrated. In the flowchart, the processing mainly performed by the server 70 is represented by (S), and the processing mainly performed by the AR display device 30 of the vehicle 90 is represented by (D).

[0084] Fig. 9 The illustrated flow is started when the vehicle 90 arrives at any of the theme parks 12 to 18, in other words, when the vehicle 90 takes any of the theme parks 12 to 18 as the departure point. Figure 1 , when the vehicle 90 stops at the stop 95 in front of each theme park 12 to 18, the process starts.

[0085] In the AR display device 30 of the vehicle 90, the position estimating unit 52 (see Figure 6) estimates the current position, and transmits information of the theme park where the vehicle 90 stops among the theme parks 12 to 18, that is, the departure theme park, from the transmission unit 51 to the server 70 (S10). The departure theme park information includes, for example, text information of the departure theme park.

[0086] The receiving unit 83 of the server 70 (see Figure 5 ) Upon receiving the information of the departure theme park, the information is sent to the route search unit 84 and the 3D model extraction unit 85. The route search unit 84 sets the destination park with reference to the park map data stored in the facility map storage unit 81 (S12). The destination theme park may be, for example, the theme park closest to the departure theme park among the theme parks 12 to 18, or the theme park farthest from the departure theme park among the theme parks 12 to 18.

[0087] Next, the route search unit 84 generates a route from the departure theme park to the destination theme park on the connecting passage 20 (refer to Figure 1 ) is the guidance route. The created guidance route is sent to the AR display device 30 via the sending unit 86 (S14).

[0088] On the other hand, the 3D model extraction unit 85 that has received the information of the departure theme park sets the character set for the departure park as a display virtual object by referring to the theme park-specific character storage unit 82. In the case where there are multiple characters pre-set for the departure park, for example, all of the multiple characters are selected as characters (virtual objects) of the departure group. The 3D model data of the virtual object representing the character of the departure group set as a display virtual object is sent to the AR display device 30 via the sending unit 86 (S16).

[0089] The receiving unit 54 of the AR display device 30 that has received the information of the guide route and the 3D model data of the display virtual object transmits the guide route information to the navigation function unit 53. In addition, the receiving unit 54 transmits the 3D model data of the display virtual object to the virtual object data storage unit 55 (S18).

[0090] exist Fig.10 , a flowchart of the display process of the augmented reality image after the above initial setting is illustrated. In this flowchart, all processing (steps) are performed by the AR display device 30. In addition, this flowchart is repeatedly executed until the departure park arrives at the destination park. The image recognition unit 57 obtains the captured image from the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the outside camera 33C (S20).

[0091] Furthermore, the image recognition unit 57 performs image recognition on the captured image (S22). In this image recognition, the image recognition unit 57 recognizes a person in the captured image (S24). For example, the learned model storage unit 58 stores teacher data of images of people in a seated state and / or images of people walking, and the neural network learned using the teacher data is used for the image recognition performed by the image recognition unit 57.

[0092] If the image is not identified as containing a person, Fig.10 The process returns to the starting point, and the in-car display unit 41A and the window display unit 41B display the captured image of the real world. On the other hand, if it is recognized that the captured image contains a person, the image recognition unit 57 performs segmentation to distinguish the image area of ​​the person in the captured image, and performs annotation to interpret that the image area represents a person. The captured image after the image recognition process is sent to the display control unit 40.

[0093] The display control unit 40 obtains the replacement ratio map (see FIG. Figure 7 ). Furthermore, the display control unit 40 obtains the guidance route information from the navigation function unit 53 and obtains the current position of the vehicle 90 from the position estimation unit 52 .

[0094] Furthermore, the display control unit 40 obtains a replacement ratio based on the current position. For example, the display control unit 40 obtains the ratio of the distance from the departure park to the current position to the distance from the departure park to the destination park, and obtains the replacement ratio corresponding to the distance ratio from the replacement ratio map (S26).

[0095] Next, the display control unit 40 selects an image area to be replaced with a virtual object image from among the plurality of image areas in which a person is recognized by the image recognition unit 57 based on the replacement ratio (S28). In addition, the display control unit 40 selects a virtual object to be replaced from the departure location group (S30). When the virtual objects in the departure location group stored in the virtual object data storage unit 55 involve a plurality of types, the display control unit 40 selects a character of the virtual object at random, for example.

[0096] Alternatively, the character of the virtual object to be replaced may be selected according to the time period. For example, the display control unit 40 confirms the current time, and if it is daytime, a diurnal animal character is selected as the virtual object to be replaced. By performing such character selection according to the time period, a performance based on the real ecology of the animal can be realized.

[0097] Furthermore, the display control unit 40 replaces the image area of ​​the selected person with a virtual object image of a character of the departure park ( S32 ).

[0098] exist Fig.11The image replacement process is illustrated in FIG. The passenger image area 101 is deleted from the captured image, and the virtual object image 102 is inserted into the deleted area. At this time, for example, a so-called hiding process such as hiding a part of the armrest of the seat by the right hand of the virtual object (penguin) can also be performed.

[0099] In addition, since the passenger image area 101 and the virtual object image 102 are different in shape and area, even if the virtual object image is inserted into the captured image, a missing portion of the image will be generated. Therefore, the display control unit 40 extracts the background image 104 (shown by hatching) that fills the missing portion from the basic image 103 and inserts it into the captured image being processed. The basic image 103 is, for example, a captured image of the interior of the vehicle and / or the connecting passage 20 without passengers, and is stored in the storage unit of the AR display device 30.

[0100] The display control unit 40 displays the augmented reality image formed as above, in which the characters are replaced with the characters of the theme park at the departure point, on the in-vehicle display unit 41A and the window display unit 41B (S34). In this way, according to the display system involved in this embodiment, since the characters set for the theme park at the departure point are displayed in the virtual reality image, services based on the world view of the theme park can be provided even after leaving the theme park.

[0101] <Augmented reality image display process of virtual objects showing the departure and destination groups>

[0102] In the above-mentioned augmented reality image display process, virtual objects of the departure group are superimposed on the image of the real world. In addition, virtual objects of a group of virtual objects set for the destination park (hereinafter appropriately referred to as the "destination group") can also be superimposed on the image of the real world.

[0103] For example, Fig.12 , an example of display of an augmented reality image when the departure park is an aquarium and the destination park is set as a zoo is shown. In this figure, a virtual object image 102 (penguin) of the departure group is displayed on the first display surface 41A1 of the in-vehicle display unit 41A, and a virtual object image 105 (bear) of the destination group is displayed on the window display unit 41B. Such image data of the virtual object of the departure group and image data of the virtual object of the destination group are stored in the theme park-specific character storage unit 82 of the server 70.

[0104] When displaying such a virtual object, the replacement ratio storage unit 56 stores the following information: Fig.13 The replacement ratio map shown in the example. In this map, except Figure 7In addition to the characteristic line L1 showing the replacement ratio of the virtual object in the departure place group, a characteristic line L2 showing the replacement ratio of the virtual object in the destination group is also drawn. For example, the replacement ratios obtained from the characteristic lines L1 and L2 can be set so that the total is 100% regardless of the location.

[0105] On the characteristic line L2, for example, the replacement ratio at the departure park is set to 0%, and the replacement ratio at the destination park is set to 100%. In addition, they increase linearly toward the destination park.

[0106] According to the characteristic line L2, the closer to the destination park, the higher the ratio (replacement ratio) of the image area of ​​the characters replaced by the virtual objects of the destination group to the image areas of all characters recognized by the image recognition unit 57. By setting such a replacement ratio, it is possible to realize a performance in which more characters of the theme park greet passengers as they approach the destination park.

[0107] exist Fig.14 In the example of using Fig.13 FIG. 1 is a flowchart of an augmented reality image display process, especially an initial setting process, in the case of a replacement scale mapping. Fig. 9 The steps shown are steps with the same number, and since the contents are the same, the description will be appropriately omitted.

[0108] In this process, on the server 70 (refer to Figure 5 ), the path search unit 84 sends the guidance path information to the sending unit 86 and also to the 3D model extraction unit 85. The 3D model extraction unit 85 selects the virtual objects of the departure group (S16) and selects the virtual objects of the destination group (S17). The image data of the selected virtual objects of the departure group and the destination group are stored in the virtual object data storage unit 55 from the sending unit 86 through the receiving unit 54 of the AR display device 30.

[0109] exist Fig.15 The flowchart of the display process of the augmented reality image after the above initial setting is illustrated in FIG. Fig.10 The steps shown are steps with the same number, and since the contents are the same, the description will be appropriately omitted.

[0110] In step S24, the image recognition unit 57 (see Figure 6) When a person is recognized in the captured images obtained by the first in-vehicle camera 33A, the second in-vehicle camera 33B, and the outside-vehicle camera 33C (S24), the display control unit 40 obtains the replacement ratio of the departure group and the destination group based on the current position (S46). Furthermore, in step S28, when the image area to be replaced with the virtual object image is selected based on the respective replacement ratios, the display control unit 40 selects the virtual object to be replaced from the departure group and the destination group (S50). As described above, in the case where the virtual objects of the departure group and the destination group stored in the virtual object data storage unit 55 involve multiple types, the display control unit 40 can also select the virtual object (character) according to the time period.

[0111] <Example of constructing a display system using a display device itself>

[0112] In the above-described embodiment, the display system is constituted by the AR display device 30 and the server 70 , but the display system disclosed in this specification is not limited to this embodiment. For example, the function of the server 70 may also be mounted on the AR display device 30 .

[0113] In the above-described embodiment, the main roles of the server 70 are to determine the destination park based on the departure park and create a guidance route, and to extract a group of virtual objects based on the departure park and the destination park.

[0114] Here, for example, when the vehicle 90 is operated as follows Fig.16 In the illustrated loop operation, the departure point and the destination are predetermined (set as fixed values), so there is no need to let the server 70 select the destination. Fig. 9 , Fig.14 In the initial setting of , instead of having the server 70 execute a part of the processing (steps), the AR display device 30 may execute all the processing.

[0115] exist Fig.17 , the functional blocks of the AR display device 30 capable of displaying an augmented reality image independently (without the server 70). Figure 5 , Figure 6 The theme park specific character storage unit 82 and the facility map storage unit 81 provided in the server 70 are stored in Fig.17 is set in the AR display device 30. Thus, for example, in Fig. 9 , Fig.14 In the initial setting process of , all processing (steps) can be performed by the AR display device 30. In addition, after the initial setting, Fig.10 , Fig.15In the illustrated display process of the augmented reality image, all the processing (steps) can be performed by the AR display device 30 alone. In addition, in this case, the display control unit 40 of the AR display device 30 can take over the 3D model extraction unit 85 ( Figure 5 ) function.

Claims

1. A display system comprising: A display device including a display unit and a display control unit, capable of moving together with a user, wherein the display control unit is capable of causing the display unit to display an augmented reality image obtained by superimposing an image of a virtual object on a scene in the real world; as well as A server is wirelessly connected to the display device. When the display device uses a facility constructed based on a specific theme as a departure point, the display device transmits facility information of the departure point to the server. The server has: a storage unit storing information of roles set for facilities at the departure point; and a transmitting unit that uses a character set for the facility at the departure point as a virtual object at the departure point and transmits image data of the virtual object to the display device, The display device is mounted on a vehicle. The display unit is arranged in a manner to separate a plurality of seats in the vehicle interior, and a first display surface and a second display surface opposite to the first display surface are respectively oriented toward the side of the vehicle interior. The display device comprises: a first camera that captures the second display surface side of the vehicle interior, and displays the captured image on the first display surface; a second camera for photographing the first display surface side of the vehicle interior, and displaying the photographed image on the second display surface; as well as an image recognition unit that performs image recognition of images captured by the first camera and the second camera, When the image recognition unit recognizes a person in the captured image, the display control unit generates the captured image by replacing the image area of ​​the recognized person with a virtual object of the departure point, and displays the captured image on the first display surface and the second display surface.

2. The display system according to claim 1, The display control unit reduces the ratio of the image area of ​​the person replaced by the virtual object of the departure place to the image areas of all recognized persons as the distance from the facility of the departure place increases.

3. The display system according to claim 2, The storage unit stores information on roles set for a facility as a destination. The transmitting unit uses a character set for the facility at the destination as a virtual object at the destination and transmits image data of the virtual object to the display device. The display control unit increases the ratio of the image area of ​​the person replaced by the virtual object of the destination relative to the image areas of all recognized persons as the destination facility approaches.

4. The display system according to claim 3, In the storage unit, a plurality of characters are set as the virtual object of the departure point and the virtual object of the destination, respectively. The display control unit selects a virtual object character to be replaced according to a time period.

5. A display device, which is a display device mounted on a vehicle, comprising: a storage unit storing information on roles set for a facility constructed based on a specific theme; an extracting unit that extracts a character set for a facility set as a departure point as a virtual object of the departure point; A display unit, wherein the display surface faces the vehicle interior; as well as a display control unit capable of causing the display unit to display an augmented reality image obtained by superimposing an image of a virtual object of the departure point on a scene in the real world, The display unit is arranged in a manner to separate a plurality of seats in the vehicle interior, and a first display surface and a second display surface opposite to the first display surface are respectively oriented toward the side of the vehicle interior. The display device further comprises: a first camera that captures the second display surface side of the vehicle interior, and displays the captured image on the first display surface; a second camera for photographing the first display surface side of the vehicle interior, and displaying the photographed image on the second display surface; as well as an image recognition unit that performs image recognition of images captured by the first camera and the second camera, When the image recognition unit recognizes a person in the captured image, the display control unit generates the captured image by replacing the image area of ​​the recognized person with a virtual object of the departure point, and displays the captured image on the first display surface and the second display surface.

Images