Image display device and image display method

Abstract

To provide an image display device and image display method, which allow for displaying user-specific information superimposed on a real space.SOLUTION: An image display device is provided, comprising an image display unit configured to cause a display device to display a superimposing image including a virtual image in such a way that the virtual image is superimposed on a scenery, an area detection unit configured to detect a display area for displaying the virtual image in the superimposing image from the scenery, and a user information acquisition unit configured to acquire user information on a user associated with the display device. The image display unit causes the display device to display a virtual image corresponding to the user information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method. [Background technology]

[0002] An advertising system using an advertising vehicle has been known for some time (see Patent Document 1). The advertising vehicle includes a body attached to cover the entire vehicle and a planar advertising unit formed on the surface of the body. The advertising system also includes an imaging unit for capturing images or videos, a detection unit for detecting coordinate points indicating spatial coordinate positions captured in the image or video captured by the imaging unit, an analysis unit for identifying spatial coordinates and three-dimensional shapes within the image or video based on the coordinate points detected by the detection unit, a synthesis unit for synthesizing computer graphics into the image or video by matching them with the spatial coordinates and three-dimensional shapes identified by the analysis unit, and a display unit for displaying the image or video synthesized by the synthesis unit. In this advertising system, multiple coordinate points are displayed on the exterior surface of the body or advertising unit of the advertising vehicle, and advertising content such as text data such as the name of a trading company or company name, or 3D data, is distributed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-181842 Summary of the Invention [Problem to be solved by the invention]

[0004] Because advertising content in conventional technology does not change according to the user's characteristics or situation, there is a problem with conventional technology in that it is not possible to superimpose information that changes according to the user's characteristics or situation onto the real space of the advertising vehicle.

[0005] An object of the present invention is to provide an image display device and an image display method that are capable of superimposing and displaying information according to a user on real space. [Means for solving the problem]

[0006] The present invention solves the above problem by detecting, from the scenery, a display area in which a virtual image is displayed in a superimposed image displayed on a display device, obtaining user information about the user associated with the display device, and displaying a superimposed image including the virtual image on the display device so that the virtual image corresponding to the user information is superimposed on the scenery. [Effects of the Invention]

[0007] According to the present invention, information according to a user can be superimposed and displayed on real space. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of an image display system according to the first embodiment. [Figure 2] FIG. 2 is a diagram for explaining the function of the controller shown in FIG. [Figure 3] FIG. 3 is a diagram for explaining a display area, which is a display target area of ​​a virtual image. [Figure 4] FIG. 4 is a diagram for explaining the display position and size of an advertisement image in a superimposed image displayed by the non-transmissive display device according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of an advertising image displayed on the non-transmissive display device shown in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of the procedure of an image display method executed by the controller shown in FIG. [Figure 7] FIG. 7 is a schematic diagram showing the configuration of an image display system according to the second embodiment. [Figure 8]FIG. 8 is a diagram for explaining the display position and size of an advertisement image in a superimposed image displayed by a transmissive display device according to the second embodiment. [Figure 9] FIG. 9 is a schematic diagram showing the configuration of an image display system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image display device and an image display method according to the present invention will be described below with reference to the accompanying drawings.

[0010] First Embodiment FIG. 1 is a schematic diagram of an image display system 100 according to a first embodiment. As shown in FIG. 1, the image display system 100 is a system mounted on a vehicle 10. In the example of FIG. 1, two people (users A and B) are riding in the vehicle 10. The image display system 100 according to this embodiment includes a mobile device 1 carried by user A, an in-vehicle device 2, and an image display device 3. The components of the image display system 100 are capable of communicating with each other using a predetermined wired communication standard (e.g., CAN) or a wireless communication standard (e.g., Bluetooth (registered trademark)), enabling information to be transmitted and received between the components. Note that the vehicle 10 is not limited to the configuration shown in FIG. 1 and may include, for example, driving devices such as an engine and a motor, an in-vehicle camera that captures images of the interior of the vehicle 10, auxiliary devices such as a sound pickup microphone and in-vehicle speakers installed in the interior of the vehicle 10, and a battery. In the following description, the term "around the current location" referring to the vicinity of the current location is used to mean within a predetermined range including the current location, and the location of the current location within the predetermined range and the size of the predetermined range are not particularly limited. For example, the size of the predetermined range can be changed as appropriate depending on the means of transportation the user is using. Also, for example, when the current location moves, the area around the current location also changes depending on the current location of the new location.

[0011] An image display device and an image display method according to the present invention are used in the field of augmented reality (AR), which is known as a technology for expanding real space by adding information. Conventionally, known methods for realizing augmented reality include a video see-through method that uses a non-transparent display device to realize augmented reality, and a see-through method that uses a transparent display device to realize augmented reality. In this embodiment, an example of the video see-through method will be described in which the image display device 3 superimposes and displays a virtual image on the display device of the mobile device 1. The virtual image is a virtual image that does not exist in real space. Furthermore, in this embodiment, a user A who views the virtual image via the mobile device 1 will be described as an example of a user who views the virtual image. Furthermore, in this embodiment, as shown in FIG. 1 , a case will be described in which, in addition to user A, user B is also riding in the vehicle 10. Users A and B may be the driver of the vehicle 10 and other passengers other than the driver. The image content of the virtual image will be described later.

[0012] The mobile device 1 is configured to include at least an imaging device (hereinafter also referred to as a mobile camera), a user interface (for example, a touch panel display, etc.), and a computer equipped with a CPU, memory, storage device, etc. The mobile device 1 is a device that can be carried by a user and is separate from the vehicle 10. The mobile camera is provided at a predetermined position on the mobile device 1 and captures images in the direction the lens is facing. In this embodiment, the images that can be captured by the mobile camera may be at least one of a still image and a video. The captured images captured by the mobile camera are output as image information to the image display device 3 at predetermined intervals. The user interface includes a non-transparent display device (hereinafter also referred to as a mobile display). Examples of the mobile device 1 include a smartphone, a tablet, etc.

[0013] For example, when user A launches an augmented reality application on mobile device 1, the mobile display functions as a non-transparent display device. A captured image of a landscape (hereinafter also referred to as a landscape image) captured by a mobile camera is output to image display device 3. Then, when processing in image display device 3, which will be described later, is completed and information on a superimposed image including the landscape image and a virtual image is input from image display device 3 to mobile device 1, the superimposed image is displayed on the mobile display. A superimposed image is an image in which a virtual image is superimposed on a landscape image that shows real space. Through the mobile display, the user feels as if the virtual image is superimposed on real space. Video see-through augmented reality allows the user to recognize real space as well as information shown by the virtual image.

[0014] The on-vehicle device 2 is a device mounted on the vehicle 10 and related to the image display system 100. As shown in FIG.

[0015] The position detection device 21 is a device for detecting the current location of the vehicle 10, and may be, for example, a GPS device. In this embodiment, for example, when the augmented reality function is enabled in the above-mentioned mobile device 1, the position detection device 21 acquires the position information of the vehicle 10 at the time when the augmented reality function was enabled. The position information of the vehicle 10 acquired by the position detection device 21 is output to the image display device 3 as the position information of the user A. Furthermore, in this embodiment, the position detection device 21 acquires the position information of the vehicle 10 at predetermined intervals, and outputs the position information of the vehicle 10 to the route guidance device 22.

[0016] The route guidance device 22 is a device for providing route guidance to the occupants of the vehicle 10, and may be, for example, a car navigation device. Position information of the vehicle 10 is input to the route guidance device 22 from the position detection device 21 at predetermined intervals. The route guidance device 22 generates a driving route for the vehicle 10 from the current location of the vehicle 10 to a destination of the vehicle 10 (for example, a destination set by the occupant), and guides the occupant along the generated driving route. For example, the route guidance device 22 guides the occupant along the driving route of the vehicle 10 by voice via an in-vehicle speaker, or guides the occupant along the driving route of the vehicle 10 by a screen display via an in-vehicle display provided in the route guidance device 22. The driving route of the vehicle 10 generated by the route guidance device 22 is output to the image display device 3 as transportation means information indicating the transportation means of the user A.

[0017] The communication device 23 is a device having a wireless communication function for wirelessly transmitting and receiving information to and from the outside of the vehicle 10, and an example of such a device is a telematics control unit (TCU). The communication device 23 connects to the Internet using a communication standard such as 4G / LTE or Wifi (registered trademark) through the wireless communication function, and transmits and receives various information to and from a server or system provided outside the vehicle 10. In this embodiment, the communication device 23 acquires information about the weather around the current location of the vehicle 10 (hereinafter also referred to as weather information) and information about the current time (hereinafter also referred to as time information) from outside the vehicle 10. The weather information and time information acquired by the communication device 23 are output to the image display device 3.

[0018] The database 24 stores various information related to the image display system 100. In this embodiment, the database 24 stores user information, advertisement information, and vehicle information, each of which belongs to a different information category. The user information indicates at least one of the user's characteristics and circumstances. Examples of the user information include gender information indicating the user's gender, age information indicating the user's age, preference information indicating the user's preferences, personal identification information that identifies the user, and a purchase history of goods or services purchased by the user via the Internet (hereinafter also referred to as purchase history information). In this embodiment, the database 24 may store at least one of the above-mentioned various types of information as user information. Various types of information are output from the database 24 to the image display device 3 in response to a request signal from the image display device 3. For example, the user inputs each item of user information into the mobile device 1 in advance, and the user information is output from the mobile device 1 to the database 24, and the user information is stored in the database 24.

[0019] The advertisement information is information used by the image display device 3 to present advertisements to users. In this embodiment, the advertisement information includes advertisement location information based on map information (hereinafter also referred to as advertisement location information) and information on advertisement content (hereinafter also referred to as advertisement content information). The advertisement location information includes location information of existing billboard advertisements and location information of markers for displaying virtual images using augmented reality (hereinafter also referred to as AR markers). The AR markers are markers for display position recognition that are used as markers to indicate the display position of virtual images in the function of the image display device 3 described below. Examples of billboard advertisements include rooftop billboards, wall billboards, protruding billboards, utility pole advertisements, and outdoor billboards. In this embodiment, an outdoor tea ceremony billboard installed on a road to promote, guide, and guide stores, companies, etc. to passersby and passengers of the vehicle 10 will be used as an example. The advertisement content information is information on a virtual image superimposed on real space and displayed by the image display device 3. The virtual image includes graphic images, text images, and images combining these. In this embodiment, an advertisement image used by a store for advertising will be used as an example of a virtual image. For example, the advertising image is acquired in advance from outside the vehicle 10 by the communication device 23 and then stored in the database 24 .

[0020] The vehicle information is information indicating the state of the vehicle 10, and in this embodiment includes information indicating the remaining amount of power in the drive source of the vehicle 10 (the remaining amount of gasoline in the case of a gasoline-powered vehicle or a hybrid vehicle, or the remaining amount of battery in the case of an electric vehicle). The remaining amount information of the drive source of the vehicle 10 stored in the database 24 is updated at a predetermined interval by an on-board computer (not shown) that manages the remaining amount of the drive source.

[0021] Next, the image display device 3 will be described. The image display device 3 is configured by a computer equipped with hardware and software, and is composed of a ROM (Read Only Memory) storing programs, a CPU (Central Processing Unit) that executes the programs stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. As shown in Fig. 1, the image display device 3 includes a controller 31. The controller 31 corresponds to the computer described above.

[0022] Fig. 2 is a diagram for explaining the functions of the controller 31 shown in Fig. 1. As shown in Fig. 2, the controller 31 includes an advertisement information acquisition unit 32, a user position information acquisition unit 33, a user information acquisition unit 34, a companion recognition unit 35, a display area detection unit 36, a display information determination unit 37, and an image display unit 38. These blocks realize the respective functions described below by software established in the ROM.

[0023] The advertisement information acquisition unit 32 acquires advertisement information including advertisement position information and advertisement content information from the in-vehicle device 2. In this embodiment, the advertisement information acquisition unit 32 extracts advertisement position information indicating that the advertisement position is near the current location of the vehicle 10 from the advertisement position information stored in the database 24. The advertisement information acquisition unit 32 also extracts advertisement content information of stores located near the current location of the vehicle 10 from the advertisement content information stored in the database 24. The advertisement information acquired by the advertisement information acquisition unit 32 is output to the display information determination unit 37.

[0024] The user position information acquisition unit 33 acquires user position information from the in-vehicle device 2. In this embodiment, the user position information acquisition unit 33 acquires the current location of the vehicle 10 from the position detection device 21 as information on the current location of user A. The user position information acquisition unit 33 also acquires weather information around the current location of the vehicle 10 and information on the current time via the communication device 23. The user position information acquired by the user position information acquisition unit 33 is output to the display information determination unit 37. Note that in this embodiment, the user position information acquired by the user position information acquisition unit 33 will be described using information on the current location of user A, weather information around the current location of user A, and information on the current time as examples, but it is sufficient that the user position information includes at least the position information of user A. Furthermore, the source from which the user location information acquisition unit 33 acquires the location information of user A is not limited to the location detection device 21. For example, if the mobile device 1 has a function equivalent to that of the location detection device 21 (for example, a GPS function), the user location information acquisition unit 33 may acquire the current location of the mobile device 1 from the mobile device 1 as information on the current location of user A.

[0025] The user information acquisition unit 34 acquires user information associated with the display device. In this embodiment, the user information acquisition unit 34 acquires user information about user A, who carries the mobile device 1, from the in-vehicle device 2. The user information acquisition unit 34 acquires the gender, age, preferences, personal identification information, and purchase history information of user A stored in the database 24, and also acquires the travel route of the vehicle 10 from the route guidance device 22. The user information acquired by the user information acquisition unit 34 is output to the display information determination unit 37. Note that in this embodiment, the user information acquired by the user information acquisition unit 34 will be described using examples of gender information of user A, age information of user A, preference information of user A, personal identification information of user A, purchase history information of user A, and transportation information of user A. However, the user information may include at least one of gender information, age information, preference information, personal identification information, purchase history information, and transportation information. Furthermore, the source from which the user information is acquired by the user information acquisition unit 34 is not limited to the in-vehicle device 2. For example, if user information is registered in the mobile device 1, the user information acquisition unit 34 may acquire the user information from the mobile device 1. Furthermore, for example, if registration of a user account including user information is required to use an augmented reality application, the user information acquisition unit 34 may acquire the user information via the communication device 23 from a server that manages accounts for the augmented reality application.

[0026] Furthermore, the user information acquiring unit 34 acquires registration information (user ID such as an account) of the portable device 1 from the portable device 1 in order to identify user A who carries the portable device 1. Then, the user information acquiring unit 34 searches for and acquires user information about user A registered in the portable device 1 from the information stored in the database 24. This allows the user information acquiring unit 34 to acquire user information about user A associated with the portable device 1, rather than user B. Furthermore, the user information acquiring unit 34 specifies user A, who is registered in the portable device 1, as a presentation target user to which the superimposed image is to be presented.

[0027] The companion recognition unit 35 recognizes a companion located within a predetermined range from the presentation target user recognized by the user information acquisition unit 34. In this embodiment, the companion recognition unit 35 acquires a captured image showing the interior of the vehicle 10 from the in-vehicle camera, and performs image recognition processing on the captured image to determine whether the presentation target user has a companion. The companion presence / absence result determined by the companion recognition unit 35 is output to the display information determination unit 37. In the example of FIG. 1 , the companion recognition unit 35 recognizes user B as a companion of the presentation target user (user A) from the captured image showing the interior of the vehicle 10, and determines that a companion is present.

[0028] The display area detection unit 36 ​​detects, from the scenery viewed by the user, a display area in which a virtual image is displayed in a superimposed image displayed by a non-transparent display device. The display area detection unit 36 ​​acquires image information from an imaging device corresponding to the type of display device, and performs image recognition processing on the acquired image information. For example, in this embodiment, when a user launches an augmented reality application on the mobile device 1, the display area detection unit 36 ​​acquires a captured image (a captured image showing the scenery in the direction the mobile camera is pointed) from the mobile camera of the mobile device 1. The display area detection unit 36 ​​also acquires advertising location information from the database 24, which indicates the location of an advertising billboard or an AR marker around the current location of the vehicle 10.

[0029] Next, a method for detecting a display area by the display area detection unit 36 ​​will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining a display area, which is a display target area of ​​a virtual image. FIG. 3 is an example of a captured image captured by a mobile camera. The captured image shows a view ahead of the vehicle 10. For example, as shown in FIG. 3, assume that there is a freestanding signboard A1 (hereinafter simply referred to as signboard A1) and a signboard M1 for an AR marker (hereinafter simply referred to as signboard M1) ahead of the vehicle 10. An advertising board A2 for displaying an advertisement A by a store is provided on the signboard M1. An AR marker board M2 for displaying an AR marker M is also provided on the signboard M1.

[0030] In the example of FIG. 3, the display area detection unit 36 ​​performs image recognition processing on the captured image to detect a display area in the entire superimposed image where a virtual image is displayed. For example, the display area detection unit 36 ​​first recognizes a signboard A1 in the captured image and detects the position of an advertisement board A2 on the signboard A1 (the height of the advertisement board A2 on the captured image). The display area detection unit 36 ​​then detects an area of ​​a size occupied by the advertisement board A2 as the display area where the virtual image is displayed. Also, for example, the display area detection unit 36 ​​first recognizes a signboard M1 in the captured image and detects the position of an AR marker board M2 on the signboard M1 (the height of the AR marker board M2 on the captured image). The display area detection unit 36 ​​then identifies the AR marker board M2 and detects the AR marker M as the display area where the virtual image is displayed. In the example of FIG. 3, as a result of processing by the display area detection unit 36, an area corresponding to the position and size of advertisement A and an area corresponding to the position and size of AR marker M are each detected as display areas. Information about the display area detected by the display area detection unit 36 ​​is output to the display information determination unit 37 and the image display unit 38 shown in FIG. 2. Note that the image recognition process performed by the display area detection unit 36 ​​may be any image recognition process known at the time of filing of the present application. The display area detection unit 36 ​​also executes the display area detection process at predetermined intervals. This allows the size and position of the display area in the captured image to be detected even if the positions and / or sizes of the signs A1 and B shown in the captured image change as the vehicle 10 moves, for example.

[0031] Returning to Fig. 2, the following describes the functions of the controller 31. The display information determination unit 37 determines the information to be displayed on the non-transmissive display device (hereinafter also referred to as display information) based on at least one of the user's characteristics and the user's situation, and generates a virtual image according to the user based on the determination result.

[0032] 2, the display information determination unit 37 receives advertisement information from the advertisement information acquisition unit 32, user location information from the user location information acquisition unit 33, user information from the user information acquisition unit 34, a companion presence / absence result from the companion recognition unit 35, and display area information from the display area detection unit 36. In this embodiment, the display information determination unit 37 determines the user's characteristics and situation from the various pieces of input information, and selects an advertisement image according to the user's characteristics and situation. Examples of the selected advertisement image will be described later.

[0033] The image display unit 38 displays a superimposed image including the virtual image on the display device so that the virtual image according to the user information input from the display information determination unit 37 is superimposed on the scenery viewed by the user. In this embodiment, for example, when user A starts an augmented reality application on the mobile device 1, the image display unit 38 displays a superimposed image including the advertisement image and the scenery image on the mobile display so that the advertisement image according to the user information of user A is superimposed on the scenery image captured by the mobile camera.

[0034] FIG. 4 is a diagram illustrating the display position and size of an advertisement image in a superimposed image displayed on a mobile display (non-transparent display device). FIG. 4 is a diagram illustrating an appearance in which a view ahead of the vehicle 10 shown in FIG. 3 is displayed on the mobile display of the mobile device 1. As shown in FIG. 4, in the scenery image showing the view ahead of the vehicle 10 shown in FIG. 3, an advertisement image B1 of advertisement B is displayed at a position corresponding to advertisement A in FIG. 3, and an advertisement image C1 of advertisement C is displayed at a position corresponding to AR marker M in FIG. 3. Because the advertisement images B1 and C1 are displayed at the positions of the advertisements that user A views via the mobile display in real space, user A can view the advertisement images B1 and C1 without feeling uncomfortable with the display positions of the advertisement images. Furthermore, because the advertisement images B1 and C1 are displayed in a size that corresponds to the size of the advertisements that user A views via the mobile display in real space, user A can view the advertisement images B1 and C1 without feeling uncomfortable with the sizes of the advertisement images B1 and C1.

[0035] FIG. 5 is a diagram illustrating an example of an advertising image displayed on the mobile display shown in FIG. 4. In FIG. 5, advertisement A on the left side is an advertisement on an existing advertising billboard, and corresponds to advertisement A shown in FIG. 3, for example. By the processing of the controller 31 described above, different advertising images are displayed for advertisement A shown in FIG. 3 according to user information. Note that users D to F shown in FIG. 5 are users who are carrying mobile device 1 and riding in vehicle 10, like user A shown in FIG. 1.

[0036] For example, if the user information of user D indicates "gender: male, age: 40, accompanied by someone, means of transportation: car" and the current time is around noon, the controller 31 determines, as the display information, information about restaurants where a man can have lunch alone. Then, the controller 31 displays an advertisement image D1 for a ramen shop located near the current location of user D at the position of advertisement A shown in FIG. 3. For example, if the user information of user E indicates "gender: female, age: 35, accompanied by someone, means of transportation: car," the current time is around 3:00 p.m., and the weather is raining, the controller 31 determines, as the display information, information about restaurants where a woman can take a break with a companion (see user B shown in FIG. 1). Then, the controller 31 displays an advertisement image E1 for a coffee shop located near the current location of user E at the position of advertisement A shown in FIG. 3. Furthermore, for example, if the user information of user F indicates "gender: female, age: 25, accompanied, means of transportation: vehicle" and the remaining amount of power source (remaining amount of gasoline) of vehicle 10 is less than a predetermined amount, controller 31 determines information about gas stations for refueling as display information. Then, controller 31 displays advertisement image F1 of gas stations located near user F's current location at the position of advertisement A shown in FIG. 3. In this way, even if the advertisement images are displayed at the same position as advertisement A, different advertisement images D1 to F1 are displayed depending on the user's characteristics and situation.

[0037] FIG. 6 is a flowchart showing an example of the steps of an image display method executed by the controller 31 shown in FIG. 1. In step S1, the controller 31 determines whether or not to present AR information. AR information is information that does not exist in real space, and corresponds to an advertising image in this embodiment. For example, when the controller 31 receives from the portable device 1 an operation signal of user A indicating that user A has launched an augmented reality application of the portable device 1, the controller 31 determines to present the AR information. On the other hand, when the controller 31 cannot receive the operation signal of user A from the portable device 1, the controller 31 determines not to present the AR information. If the controller 31 makes a positive determination, the process proceeds to step S2; if the controller 31 makes a negative determination, the controller 31 ends the process shown in FIG. 6.

[0038] In step S2, the controller 31 acquires various pieces of information necessary to display the AR information. In this embodiment, the controller 31 acquires advertisement information, user location information, weather information, time information, and user information from the in-vehicle device 2. The controller 31 also acquires registration information of the mobile device 1 from the mobile device 1 in order to identify user A. The controller 31 also acquires captured images of the interior of the vehicle 10 from the in-vehicle camera in order to determine whether or not user A is accompanied by someone.

[0039] In step S3, the controller 31 detects a display area in which the advertisement image is displayed in the superimposed image displayed on the non-transparent display device, based on the advertisement position information acquired in step S2. For example, the controller 31 extracts billboard advertisements or AR markers installed around the current location of the user A. The controller 31 also recognizes the billboard advertisements or AR markers included in the captured image by performing image processing on the captured image captured by the mobile device 1. The controller 31 detects the area of ​​the billboard advertisement included in the captured image as the display area. The controller 31 also detects the area of ​​the AR marker included in the captured image as the display area.

[0040] In step S4, the controller 31 determines display information to be displayed on the non-transparent display device based on the advertisement information, user location information, and user information acquired in step S2, and the display area detected in step S3. For example, as described with reference to Fig. 5, the controller 31 determines the display information based on the user's gender, age, preferences, whether or not the user is accompanied, and means of transportation (including vehicle information). In this embodiment, the controller 31 selects an advertisement image based on the determined display information.

[0041] In step S5, the controller 31 generates a superimposed image to be displayed on the non-transmissive display device based on the display information determined in step S4. In this embodiment, the controller 31 generates a superimposed image including a scenery image captured by the mobile camera and the advertisement image selected in step S4. The controller 31 generates the superimposed image so that the advertisement image is displayed in the display area detected in step S3.

[0042] In step S6, the controller 31 outputs the superimposed image generated in step S5 to the non-transparent display device. In this embodiment, the controller 31 causes the non-transparent display device to display the superimposed image including the advertisement image and the scenery image so that the advertisement image according to the user information is superimposed on the scenery image.

[0043] In step S7, the controller 31 determines whether or not to continue presenting the AR information. This step corresponds to step S1, and therefore the description of step S1 is used for the processing content by the controller 31. If the controller 31 makes a positive determination, the process returns to step S2, and if the controller 31 makes a negative determination, the controller 31 ends the processing shown in Fig. 6. After returning to step S2, the processes of steps S2 to S6 are repeatedly executed until a negative determination is made in step S7, and the AR information continues to be displayed on the non-transmissive display device.

[0044] As described above, the image display device 3 according to this embodiment includes an image display unit 38 that displays a superimposed image including an advertising image on a non-transparent display device so that the advertising image is superimposed on a landscape, a display area detection unit 36 ​​that detects a display area in the superimposed image where the advertising image is to be displayed from a landscape image captured by a mobile camera, and a user information acquisition unit 34 that acquires user information related to the user associated with the non-transparent display device. The image display unit 38 displays a superimposed image including the advertising image and the landscape image on the non-transparent display device so that the advertising image according to the user information is superimposed on the landscape image. This allows advertising image information according to the user to be superimposed and displayed in real space.

[0045] In this embodiment, the user information indicates at least one of the characteristics and the situation of the user viewing the image displayed on the mobile display, thereby allowing the display of advertising images suited to the characteristics and the situation of the user.

[0046] In this embodiment, the user information includes at least one of the user's personal identification information, the user's means of transportation information, the user's preference information, and the user's purchase history information. This allows advertising image information to be displayed in the display area according to the user's characteristics and situation. For example, it is possible to present to the user advertising image information specialized for the user's personal characteristics, advertising image information according to the user's means of transportation and situation, advertising image information tailored to the user's preferences, advertising image information tailored to the user's purchase history, etc.

[0047] In this embodiment, the virtual image that does not exist in real space is an advertisement image, and the display area in which the advertisement image is displayed is the area of ​​a signboard in the scenery. As a result, the information of the advertisement image is displayed in the part of the existing signboard, and the user is given the illusion that the information of the advertisement image is information of the existing signboard. As a result, the discomfort felt by the user regarding the display position of the advertisement image can be reduced. Because the user can simultaneously view the real space, the information of the advertisement image can be presented to the user while ensuring the user's field of vision.

[0048] In this embodiment, the display area detection unit 36 ​​detects the display area by identifying an AR marker that indicates the display position of the virtual image from the scenery image. This allows the position and range of the display area to be detected with high accuracy. Furthermore, it also makes it easy to distinguish between the display area and other areas.

[0049] The image display device 3 according to this embodiment also includes an advertisement information acquisition unit 32 that acquires marker position information related to the positions of AR markers, and a user position information acquisition unit 33 that acquires user position information related to the user's position. The display area detection unit 36 ​​identifies AR markers located within a predetermined range from the user based on the marker position information and the user position information. This makes it possible to identify AR markers that are visible to the user via the mobile device.

[0050] The image display device 3 according to this embodiment also includes a companion recognition unit 35 that recognizes a companion of the user who is located within a predetermined range from the user. When a companion is recognized, the image display unit 38 causes the non-transparent display device to display an advertising image corresponding to the combination of user A and the companion user B. This makes it possible to present to the user information about advertising images that take into account the user's companion. For example, as described with reference to FIG. 5, it is possible to present to the user information about advertising images for coffee shops where the user can take a break with their companion.

[0051] In this embodiment, the user who views the superimposed image displayed on the non-transmissive display device is a passenger in the vehicle 10. This allows an advertising image suitable for the user to be displayed even when the user is traveling in the vehicle 10.

[0052] Second Embodiment FIG. 7 is a schematic diagram of an image display system 200 according to a second embodiment. In the first embodiment described above, a video see-through method using a mobile device 1 was explained, but in this embodiment, an example of a see-through method in which an image display device 3 displays a virtual image superimposed on the windshield of a vehicle 10 will be explained. The image display system 200 includes an in-vehicle device 2, a head-up display 4 configured integrally with the windshield of the vehicle 10, and the image display device 3. In FIG. 7, components similar to those in the image display system 100 are assigned the same reference numerals as those shown in FIG. 1. The above explanation will be used to refer to components similar to those in the image display system 100, and differences from the image display system 100 will be explained.

[0053] The head-up display 4 is a device that is provided, for example, at a lower position of the windshield of the vehicle 10 and displays an image on the windshield using lighting devices. In the head-up display 4, the windshield functions as a transmissive display device. The head-up display 4 is configured to include at least a computer equipped with a CPU, memory, storage device, etc.

[0054] For example, when user A sets the augmented reality function of the head-up display to be enabled, an image captured by the imaging device 25 (described later) is output to the image display device 3 as an image of the range in front of the vehicle 10 that user A can see (an image corresponding to user A's field of view). Then, when processing by the image display device 3 (described later) is completed and information on a superimposed image including a virtual image is input from the image display device 3 to the head-up display 4, the superimposed image is displayed on the windshield. User A can perceive the real space located on the opposite side of the windshield from the user A through the windshield. When the superimposed image from the image display device 3 is displayed on the windshield, user A feels as if the virtual image is superimposed on the real space. The see-through augmented reality allows user A to perceive the real space and also perceive the information displayed by the virtual image. Furthermore, in this embodiment, the see-through method uses a head-up display visible to users A and B, so not only user A but also user B can perceive the real space and perceive the information displayed by the virtual image.

[0055] As shown in FIG. 7, in this embodiment, the in-vehicle device 2 includes a position detection device 21, a route guidance device 22, a communication device 23, a database 24, and an imaging device 25.

[0056] The imaging device 25 includes an exterior camera that captures images outside the vehicle 10 and an interior camera that captures images inside the vehicle 10. The exterior camera is provided, for example, at a predetermined position in front of the vehicle 10 and captures images in front of the vehicle 10. The interior camera is provided at a predetermined position inside the vehicle 10 and captures images inside the vehicle 10. In this embodiment, the images that can be captured by the imaging device 25 may be at least one of still images and videos. The images captured by the exterior camera are output to the image display device 3 as image information at predetermined intervals. The images captured by the interior camera are output to the image display device 3 as passenger number information. The type, number, and installation location of the imaging device 25 are not particularly limited.

[0057] Next, the functions of the image display device 3 according to this embodiment that differ from the functions of the image display device 3 according to the first embodiment will be described.

[0058] In the present embodiment, the user information acquiring unit 34 acquires user information about a user viewing the head-up display 4 from the in-vehicle device 2. Furthermore, the user information acquiring unit 34 acquires registration information about the vehicle 10 (such as an ID indicating the owner of the vehicle) from the database 24 in order to identify the user viewing the head-up display 4. Then, the user information acquiring unit 34 searches for and acquires user information about the owner of the vehicle 10 from the information stored in the database 24. This allows the user information acquiring unit 34 to acquire user information associated with the head-up display 4. In the example of FIG. 7 , the user information acquiring unit 34 acquires user information about user A who is the owner of the vehicle 10, rather than user B. Furthermore, the user information acquiring unit 34 identifies user A who is the owner of the vehicle 10, as a presentation target user to which a superimposed image including a virtual image is to be presented.

[0059] Similar to the first embodiment described above, the companion recognition unit 35 according to this embodiment acquires a captured image showing the interior of the vehicle 10 from the in-vehicle camera and performs image recognition processing on the captured image to determine whether the presentation target user has a companion. In the example of Fig. 7, the companion recognition unit 35 recognizes user B as a companion of user A, who is the presentation target user, from the captured image showing the interior of the vehicle 10, and determines that a companion is present.

[0060] In this embodiment, the display area detection unit 36 ​​detects a display area in which a virtual image is displayed in the superimposed image displayed by the transmissive display device from the scenery visually recognized by the user. For example, when the user A sets the augmented reality function of the head-up display 4 to be enabled, the display area detection unit 36 ​​acquires a captured image (a captured image showing the scenery ahead of the vehicle 10) captured by the imaging device 25.

[0061] In this embodiment, the display information determination unit 37 determines the display information to be displayed on the transparent display device based on at least one of the user's characteristics and the user's situation, and selects an advertising image appropriate for the user based on the determination result.

[0062] In this embodiment, the image display unit 38 displays a superimposed image including the virtual image on the display device so that the virtual image input from the display information determination unit 37 is superimposed on the scenery viewed by the user. For example, when the user A sets the augmented reality function of the head-up display 4 to be enabled, the image display unit 38 displays a superimposed image including the advertising image on the windshield so that an advertising image according to the user information of the user A is superimposed on the scenery ahead of the vehicle 10 viewed by the user A.

[0063] FIG. 8 is a diagram for explaining the display position and size of an advertisement image in a superimposed image displayed on a windshield (transmissive display device). FIG. 8 is a diagram illustrating a state in which a view ahead of the vehicle 10 shown in FIG. 3 is displayed on the windshield. As shown in FIG. 8, with respect to the view ahead of the vehicle 10 shown in FIG. 3, an advertisement image B1 of advertisement B is displayed at a position corresponding to advertisement A in FIG. 3, and an advertisement image C1 of advertisement C is displayed at a position corresponding to AR marker M in FIG. 3. The advertisement images B1 and C1 are displayed at positions of the advertisements that user A views through the windshield in real space. Furthermore, the advertisement images B1 and C1 are displayed corresponding to the sizes of the advertisements that user A views through the windshield in real space. Therefore, it is possible to obtain the same effect as that described using FIG. 4 in the first embodiment.

[0064] As described above, the image display unit 38 according to this embodiment displays a superimposed image including an advertising image on the windshield provided in front of the vehicle 10 so that the advertising image according to the user information is superimposed on the user's field of vision through the windshield. This allows an advertising image suitable for the user to be displayed in front of the vehicle even when the user is traveling in the vehicle 10.

[0065] Third Embodiment FIG. 9 is a schematic diagram of an image display system 300 according to a third embodiment. In the second embodiment described above, a see-through method using a head-up display 4 was described. In this embodiment, however, an example of the see-through method in which an image display device 3 displays a virtual image superimposed on a display device of an AR device 5 will be described. The image display system 300 includes an in-vehicle device 2, an AR device 5 worn by a user A, and an image display device 3. In FIG. 9, components similar to those in the image display systems 100 and 200 are denoted by the same reference numerals as those shown in FIG. 1 or 7. The above-described explanations will be used for components similar to those in the image display systems 100 and 200, and differences from the image display systems 100 and 200 will be described.

[0066] The AR device 5 is a device (also referred to as a wearable device) that can be worn by a user and is a device separate from the vehicle 10. The AR device 5 is configured to include at least an imaging device (hereinafter also referred to as an AR camera), a see-through display device (hereinafter also referred to as an AR display), and a computer including a CPU, memory, a storage device, etc. The AR camera includes an outward-facing camera that captures an image in the direction the user's head is facing and an inward-facing camera that captures an image of the user's eyes. In this embodiment, the images that can be captured by the AR camera may be at least one of still images and videos. The captured images captured by the AR camera are output as image information to an image display device at predetermined intervals. An example of the AR device 5 is a glasses-shaped device (also referred to as AR glasses) that can be worn on the user's head. In this embodiment, AR glasses will be described as an example, but the shape of the AR device 5 is not limited to glasses. The shape of the AR device 5 may be, for example, a hat-shaped or helmet-shaped device that covers the user's head, or a pair of goggles that cover the user's eyes.

[0067] For example, if user A wears the AR device 5 and sets the augmented reality function to be enabled, an image captured by the outward-facing camera is output to the image display device 3 as an image corresponding to user A's field of view. Then, when processing in the image display device 3, which will be described later, is completed and information on a superimposed image including a virtual image is input from the image display device 3 to the AR device 5, the superimposed image is displayed on the AR display. Through the AR display, user A can recognize the real space located on the opposite side of the AR display from the user A. When the superimposed image from the image display device 3 is displayed on the AR display, user A feels as if the virtual image is superimposed on the real space. With the see-through augmented reality, the present embodiment can also achieve the same effects as the second embodiment using the head-up display 4.

[0068] Next, the functions of the image display device 3 according to this embodiment that differ from the functions of the image display device 3 according to the first and second embodiments will be described.

[0069] In this embodiment, the user information acquisition unit 34 acquires user information about the user viewing the AR display from the in-vehicle device 2. Furthermore, the user information acquisition unit 34 acquires registration information of the AR device 5 from the database 24 to identify the user viewing the AR display. Then, the user information acquisition unit 34 searches for and acquires user information about user A who uses the AR device 5 from the information stored in the database 24. This allows the user information acquisition unit 34 to acquire user information associated with the AR display. In the example of FIG. 9 , the user information acquisition unit 34 acquires user information about user A wearing the AR device 5, rather than user B. Furthermore, the user information acquisition unit 34 identifies user A wearing the AR device 5 as a presentation target user to which a superimposed image including a virtual image is to be presented.

[0070] In this embodiment, the display area detection unit 36 ​​and the image display unit 38 have the same functions as the display area detection unit 36 ​​and the image display unit 38 according to the second embodiment. For example, when user A is wearing the AR device 5 and the augmented reality function is set to be enabled, the display area detection unit 36 ​​acquires a captured image captured by the outward-facing camera (a captured image showing the scenery in the direction in which user A's head is facing). Furthermore, the image display unit 38 displays a superimposed image including an advertisement image on the AR display so that the advertisement image according to user A's user information is superimposed on the scenery viewed by user A.

[0071] As described above, the image display unit 38 according to the present embodiment displays a superimposed image including an advertisement image on the AR display so that the advertisement image according to the user information is superimposed on the user's field of view via the AR display, thereby allowing the user to view an advertisement image suited to the user via the AR device.

[0072] The above-described embodiments and modifications are provided to facilitate understanding of the present invention, and are not intended to limit the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

[0073] For example, in the above-described embodiment, the image display device 3 is mounted on the vehicle 10. However, the image display device 3 may be provided in a location other than the vehicle 10. For example, the image display device 3 may be provided in a server provided outside the vehicle 10 and capable of communicating with the mobile device 1, the AR device 5, or the vehicle 10. In this case, the image display device 3 exchanges information with the mobile device 1, the AR device 5, or the vehicle 10 via wireless communication. Furthermore, for example, in the above-described embodiment, a vehicle occupant is used as an example of a user viewing a virtual image. However, the user viewing a virtual image is not limited to being a passenger in a vehicle. The image display device and image display method according to the present invention can also be applied to a user positioned on a sidewalk, for example. As with the above-described embodiment, the image display device and image display method according to the present invention can be applied to a user positioned on a sidewalk, regardless of whether the user is moving or not.

[0074] Furthermore, for example, in the above-described embodiment, the case where there is one AR marker in the captured image has been described as an example, but a case where there are multiple AR markers will now be described. When there are multiple AR markers within a predetermined range from the user, the display area detection unit 36 ​​identifies each of the multiple AR markers by determining the type of AR marker. For example, the display area detection unit 36 ​​determines the type of AR marker based on identification information assigned to each AR marker. Since it is possible to identify each of the multiple AR markers and specify them as display areas, advertising image information tailored to the user can be displayed in multiple display areas.

[0075] Furthermore, for example, in the above-described embodiment, the vehicle information has been described using information on the remaining capacity of the vehicle's drive source as an example, but the vehicle information is not limited to this, and may be, for example, information on the vehicle's driving history. Furthermore, the user information acquisition unit 34 may identify the user information from the driving history of the vehicle 10 stored in the route guidance device 22. For example, the user information acquisition unit 34 identifies, as user information, categories that the user tends to search for (parking lots, restaurants, amusement parks) from search results for surrounding areas that are included in the driving history of the vehicle 10 and that the user inputs to the route guidance device 22. As a result, the display information determination unit 37 at the subsequent stage determines display information taking into account information on the categories that the user tends to search for, and therefore, advertising images for stores in the categories that the user tends to search for are displayed on the display device. [Explanation of symbols]

[0076] 1. Mobile devices 2…In-vehicle equipment 21...Position detection device 22...Route guidance device 23...Communication equipment 24...Database 25...imaging device 3...Image display device 31...Controller 32...Advertising information acquisition unit 33...User location information acquisition unit 34...User information acquisition unit 35…Companion recognition unit 36...Display area detection unit 37…Display information judgment unit 38...Image display unit 4. Head-up display 5…AR equipment 10...Vehicle 100, 200, 300...Image display system

Claims

1. an image display unit that displays a superimposed image including the virtual image on a display device so that the virtual image is superimposed on a landscape; an area detection unit that detects, from the scenery, a display area in which the virtual image is displayed in the superimposed image; a user information acquisition unit that acquires user information related to a user associated with the display device, the virtual image is an advertisement image; the display area is an area of ​​a signboard in the scene; the image display unit displays the virtual image corresponding to the user information on the display device; When the remaining amount of the driving source of the vehicle is equal to or less than a predetermined amount, the advertisement image of a facility that replenishes the remaining amount of the driving source is displayed in the display area; When the remaining amount of the driving source of the vehicle is higher than a predetermined amount, the advertising image corresponding to the user information is displayed in the display area.

2. the display device is a non-transparent display device provided in a portable device that can be carried by the user, the scenery is a scenery image of the scenery captured by the mobile device, The image display device according to claim 1 , wherein the image display unit causes the display device to display the superimposed image including the virtual image and the scenery image so that the virtual image according to the user information is superimposed on the scenery image.

3. the display device is a transmissive display device provided in a wearable device that can be worn on the user's head, the view is the view of the user through the display device, The image display device according to claim 1 , wherein the image display unit causes the display device to display the superimposed image including the virtual image so that the virtual image according to the user information is superimposed on the scenery.

4. 4. The image display device according to claim 1, wherein the user information is information indicating at least one of the characteristics and the situation of the user viewing the image displayed on the display device.

5. 5. The image display device according to claim 4, wherein the user information includes at least one of identification information for identifying the user, transportation information indicating the user's transportation means, preference information indicating the user's preferences, and purchase history information indicating the user's purchasing history.

6. 6. The image display device according to claim 1, wherein the area detection unit detects the display area by identifying a marker indicating a display position of the virtual image from the scenery.

7. a first position information acquisition unit that acquires marker position information relating to the position of the marker; a second location acquisition unit that acquires user location information relating to the location of the user; The image display device according to claim 6 , wherein the area detection unit identifies the marker located within a predetermined range from the user based on the marker position information and the user position information.

8. The image display device according to claim 7 , wherein, when a plurality of the markers are present within a predetermined range from the user, the area detection unit identifies each of the plurality of markers by determining the type of the marker.

9. a companion recognition unit that recognizes a companion of the user who is located within a predetermined range from the user; The image display device according to any one of claims 1 to 8, wherein the image display unit, when the companion is recognized, causes the display device to display the virtual image corresponding to the combination of the user and the companion.

10. 10. The image display device according to claim 1, wherein the user is a passenger of the vehicle.

11. the display device is a transmissive display device provided at the front of the vehicle, the view is the view of the user through the display device, The image display device according to claim 10 , wherein the image display unit causes the display device to display the superimposed image including the virtual image so that the virtual image according to the user information is superimposed on the scenery.

12. The image display device according to claim 10 , wherein the user information acquisition unit identifies the user information from information about the vehicle including a driving history of the vehicle.

13. An image display method in which a controller displays a superimposed image on a display device, Detecting a display area in the superimposed image from the scenery where a virtual image is displayed; obtaining user information about a user associated with the display device; displaying the superimposed image including the virtual image on the display device so that the virtual image according to the user information is superimposed on the scenery; the virtual image is an advertisement image; the display area is an area of ​​a signboard in the scene; When the remaining amount of the driving source of the vehicle is equal to or less than a predetermined amount, the advertisement image of a facility that replenishes the remaining amount of the driving source is displayed in the display area; When the remaining amount of the driving source of the vehicle is higher than a predetermined amount, the advertising image corresponding to the user information is displayed in the display area.

Images