Information processing device, display device, control device, image generation method, and program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2024-12-16
- Publication Date
- 2026-06-26
Smart Images
Figure 2026105665000001_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a process of superimposing and displaying images.
Background Art
[0002] As a device for experiencing AR (Augmented Reality) content, there is a display device such as AR glasses. According to a display device such as AR glasses, since content based on CG or the like can be displayed to the user as if it were superimposed on the real world, it becomes possible to provide the user with an extended real-world experience. The content to be superimposed on the real world varies depending on the usage and scene, such as advertisements, information, characters, etc.
[0003] Patent Document 1 describes registering content in association with an area of the real world.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] If content is registered for each area, the content corresponding to the position of the user using a display device such as AR glasses can be displayed as if it were superimposed on the real world where the user is located. By the way, in a display device such as AR glasses used by the user, in addition to the content corresponding to the area where the user is located, there may be a case where the content corresponding to the area where the user is not located is also desired to be superimposed and displayed. For example, assume that there is a main store of a clothing store and a branch store of the same brand as that clothing store. In this case, when the user is in the branch store, there may be a desire to superimpose and display not only the content corresponding to the branch store but also the content corresponding to the main store where the user is not located.
[0006] To perform such display control using the method described in Patent Document 1, it becomes necessary to associate and register the parent store's content in the parent store's area, and to associate and register the child store's content with the parent store's content in the child store's area. Therefore, if the connections between areas become complex, the administrator's work of associating and registering areas with content may become cumbersome. In addition, if an area is newly created or deleted, the administrator may need to re-associate and register the area with content, which may increase the burden on the administrator. [Means for solving the problem]
[0007] The information processing apparatus disclosed herein is an information processing apparatus that outputs a display image to a display device that processes a real world on which a display image is superimposed so that a user can see the real world, and is characterized by comprising: management means for managing content associated with each of a plurality of areas in the real world and the dependency relationships between the areas constituting the plurality of areas; and generation means for generating the display image which includes images corresponding to each of the content associated with the area of the plurality of areas that includes the location of the display device and the content associated with the area to which the area including the location of the display device is dependent. [Effects of the Invention]
[0008] The technology disclosed herein reduces the effort required to manage content when displaying area-specific content on a display device. [Brief explanation of the drawing]
[0009] [Figure 1] A diagram showing an example of the configuration of an information processing system. [Figure 2] A diagram illustrating the internal structure of AR glasses. [Figure 3] A diagram showing the hardware configuration of the information processing server and control unit. [Figure 4]Block diagram showing the functional configurations of an information processing server and a control device. [Figure 5] Flowchart for explaining the superimposed display process of a display image. [Figure 6] Flowchart for explaining the generation process of a display image. [Figure 7] Diagram showing an example of setting a display area. [Figure 8] Diagram showing an example of the dependency relationship of display areas. [Figure 9] Diagram for explaining a display image and the real world with the display image superimposed. [Figure 10] Block diagram showing the functional configurations of an information processing server and a control device. [Figure 11] Flowchart for explaining the generation process of a display image. [Figure 12] Diagram showing an example of setting a display area. [Figure 13] Diagram showing an example of the dependency relationship of display areas. [Figure 14] Diagram for explaining the real world with a display image superimposed. [Figure 15] Diagram for explaining the real world with a display image superimposed. [Figure 16] Block diagram showing the functional configurations of an information processing server and a control device. [Figure 17] Flowchart for explaining the generation process of a display image. [Figure 18] Diagram showing an example of setting a display area.
Embodiments for Carrying Out the Invention
[0010] Hereinafter, embodiments of the technology of the present disclosure will be described in detail with reference to the drawings. In the present specification and drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant descriptions are omitted.
[0011] <The First Embodiment> Figure 1 shows an example of the configuration of an information processing system according to this embodiment. The information processing system includes AR glasses 101, a control device 102, and an information processing server 103.
[0012] The AR glasses 101 are a glasses-type display device that performs display processing so that the user can see content images that do not exist in the real world superimposed on the real world. In this embodiment, the content image is described as a video (moving image), but it may also be a still image. The content image is a two-dimensional image generated from CG content. An image containing the content image is also called a display image.
[0013] The control device 102 is an example of an information processing device connected to the AR glasses 101. In this embodiment, as an example, the control device 102 is included in the AR glasses 101, and the control device 102 and the AR glasses 101 are described as an integrated device. That is, in this embodiment, the functions and processes performed by the control device 102 can also be said to be the functions and processes performed by the AR glasses 101.
[0014] Figure 2 is a diagram illustrating an example of the internal configuration of AR glasses 101. The AR glasses 101 have a display 201 such as an LCD or LCOS for displaying images. The control device 102 sends a display image to the display 201 and performs display control to display the image on the display 201.
[0015] The dotted line in Figure 2 represents the light showing the display image emitted from the display 201. The light showing the display image emitted from the display 201 passes through the lens 202 of the parallel optical system, is diffracted by the input diffracting element 203, and travels in the in-plane direction through the light guide plate 204. The light emitted from the display 201 is guided to the user's eyes by total internal reflection within the light guide plate 204, where it changes course again outside the light guide plate by the output diffracting element 205 and is guided to the user's eyes.
[0016] The solid arrows in Figure 2 represent light in the real world. As shown in Figure 2, the AR glasses 101 are configured so that light from the real world is guided to the user's eyes without being obstructed. As shown in Figure 2, the light representing the display image guided by the output diffraction element 205 and the light from the real world combine and enter the user's eyes, allowing the user to see the real world as if the display image were superimposed.
[0017] The information processing server 103 is an information processing device that generates a two-dimensional display image from three-dimensional CG content corresponding to the display target area described later, and transmits information about the generated display image to the control device 102.
[0018] The system configuration shown in Figure 1 is an example, and the configuration of the information processing system according to this embodiment is not limited. Another example of the configuration of the information processing system is the inclusion of an input device such as a controller or keyboard for receiving input from the user, and the input device may be connected to the control device 102 via a predetermined communication path. The control device 102 may also be configured to be separate from the AR glasses 101, in which case, for example, the control device 102 and the AR glasses 101 may be connected by a communication cable.
[0019] [Hardware configuration] Figure 3 illustrates an example of the hardware configuration of the devices constituting the information processing system according to this embodiment.
[0020] The control unit 102 includes a CPU (Central Processing Unit) 301, RAM (Random Access Memory) 302, and ROM (Read Only Memory) 303. Furthermore, the control unit 102 includes an HDD (Hard Disk Drive) 304, a general-purpose I / F (Interface) 305, an image output I / F 306, and a network I / F 307. The components included in the control unit 102 are connected to each other via the main bus 300, enabling them to send and receive information.
[0021] The CPU 301 is a processor that comprehensively controls each part within the control unit 102. The RAM 302 functions as the main memory, work area, etc., of the CPU 301. The ROM 303 is a storage unit that stores the programs executed by the CPU 301. The HDD 304 includes a storage area that stores applications executed by the CPU 301, data used for image processing, etc. Note that this storage area is not limited to an HDD but may be included in various storage devices. As a specific example, this storage area may be included in an auxiliary storage device such as an SSD (Solid State Drive) instead of or together with the HDD 304.
[0022] The general-purpose I / F305 is a serial bus interface compliant with standards such as USB and IEEE1394, and can be connected to, for example, an IMU (not shown) installed in the AR glasses 101 to acquire attitude information.
[0023] The image output interface 306 is an interface such as HDMI® or DisplayPort, and is used to transmit images to be displayed on the display 201 of the AR glasses 101.
[0024] The network interface 307 is an interface for connecting to a predetermined network in order to communicate with the information processing server 103. It is also used for communication via BLE, Wi-Fi, etc., when acquiring location information of the AR glasses 101. The configuration of the network interface 307 may be changed as appropriate depending on the type of network to be connected to and the communication method applied.
[0025] The information processing server 103 has a CPU 311, RAM 312, ROM 313, HDD 314, general-purpose I / F 315, image output I / F 316, and network I / F 317, and each component is connected by a bus 310 so that it can send and receive information from one another.
[0026] Since CPU311 is the same as CPU301 of control unit 102, its description is omitted. Similarly, since ROM313 is the same as ROM303, RAM312 is the same as RAM302, and HDD314 is the same as HDD304, their descriptions are omitted.
[0027] The general-purpose I / F315 is connected via USB to a keyboard and mouse for operating the information processing server 103. The image output I / F316 is connected via HDMI® or similar to a display (not shown) for displaying information such as the operation screen of the information processing server 103. The network I / F317 is an interface for information communication with the control device 102.
[0028] [Functional Configuration] Figure 4 shows an example of the configuration of the functions of the control device 102 and the information processing server 103. The control device 102 includes a location information acquisition unit 401, a communication control unit 402, and a display control unit 403.
[0029] The location information acquisition unit 401 acquires the location information of the AR glasses 101 in three-dimensional space. The method of acquiring location information is not limited. For example, known location information acquisition methods may be used depending on the location of use and required accuracy, such as using GPS, using a BLE (Bluetooth Low Energy) compatible beacon, or measuring by the signal strength of a wireless LAN access point via Wi-Fi.
[0030] The communication control unit 402 communicates with the information processing server 103 and performs transmission and reception processing such as sending location information of the AR glasses 101 to the information processing server 103 and receiving display images and other data from the information processing server 103.
[0031] The display control unit 403 processes the received display image to be displayed on the display 201 of the AR glasses 101.
[0032] The information processing server 103 includes an area dependency relationship acquisition unit 411, an entry / exit detection unit 412, a display target area determination unit 413, a content acquisition unit 414, a display image generation unit 415, a communication control unit 416, an area information acquisition unit 417, and a content management unit 418.
[0033] The area dependency relationship acquisition unit 411 acquires information indicating the dependency relationships between display areas that constitute a plurality of pre-configured display areas. The dependency relationships between display areas will be described later.
[0034] The entry / exit detection unit 412 acquires the position information of the AR glasses 101 transmitted from the control device 102 and performs a determination process to determine whether or not the AR glasses 101 have entered or exited each display area.
[0035] The content management unit 418 manages the stored CG content by associating it with display areas. The content management unit 418 also manages the dependency relationships between the configured display areas. The CG content is used to generate the display images described later.
[0036] The display target area determination unit 413 determines the CG content to be displayed based on the dependency relationships between display areas. In this embodiment, CG content is managed in association with display areas. Therefore, the display target area determination unit 413 determines the CG content to be displayed by determining the display area (referred to as the display target area) to which the CG content to be displayed is associated.
[0037] The content acquisition unit 414 acquires the CG content associated with the display target area.
[0038] The display image generation unit 415 generates a display image to be superimposed onto the real world in the AR glasses 101 from the CG content acquired by the content acquisition unit 414.
[0039] The communication control unit 416 is a receiving unit that communicates with the control device 102 and receives location information, etc., and an output unit (transmitting unit) that outputs (transmits) content information and display images, etc.
[0040] The area information acquisition unit 417 acquires information about pre-set display areas. For example, it acquires information indicating the location of each display area and information regarding the size of the area.
[0041] The functional units of the control device 102 and the information processing server 103 shown in Figure 4 are realized by the execution of predetermined programs by the CPUs of each device, but are not limited to this. Other hardware, such as a GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array), may also be used to speed up calculations. Each functional unit may be realized through the cooperation of software and hardware such as dedicated ICs, or some or all of the functions may be realized by hardware alone.
[0042] [Overlaying of displayed images] Figure 5 is a flowchart illustrating the superimposition display process performed in the control device 102. The series of processes shown in Figure 5 are realized when a program stored in the ROM 303 or HDD 304 of the control device 102 is loaded into the RAM 302 and executed by the CPU 301. Alternatively, some or all of the functions of the steps in Figure 5 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in that flowchart, and the same applies to subsequent flowcharts.
[0043] In S501, the position information acquisition unit 401 acquires position information, including information about the position of the AR glasses 101 in three-dimensional space. The position information includes not only the position of the AR glasses 101 but also information about the orientation of the AR glasses 101. The position information is acquired by a known method, as described above.
[0044] In S502, the communication control unit 402 transmits the location information of the AR glasses 101 acquired in S501 to the information processing server 103.
[0045] In S503, the communication control unit 402 receives the display image generated by the information processing server 103 and transmitted from the information processing server 103. Details of the display image generation process in the information processing server 103 will be described later.
[0046] In S504, the display control unit 403 outputs the display image acquired in S503 to the display 201 of the AR glasses 101 and processes it so that the display image is displayed on the display 201. As a result, as described above, the user wearing the AR glasses 101 can see the real world with the display image generated by the information processing server 103 superimposed on it.
[0047] In S505, the display control unit 403 determines whether or not to terminate the display. For example, if the user sends a command to terminate the display to the control device 102 using a controller (not shown), the display control unit 403 determines to terminate the display. In this case (S505 is YES), the display control unit 403 terminates the display on the AR glasses 101's display 201, and the flowchart in Figure 5 ends. After the flowchart in Figure 5 ends, the user wearing the AR glasses will only be able to see the real world. On the other hand, if the user does not send a command to terminate the display, and the display control unit 403 determines that the display will not be terminated (S505 is NO), it returns to S501 and continues processing.
[0048] [Image generation process] Figure 6 is a flowchart illustrating the display image generation process performed in the information processing server 103. The series of processes shown in Figure 6 are realized when a program stored in the ROM 313 or HDD 314 of the information processing server 103 is loaded into the RAM 312 and executed by the CPU 311.
[0049] The flowchart in Figure 6 is assumed to be executed repeatedly in real time at predetermined intervals. For example, if the display image is superimposed on the real world at 30fps, the flowchart in Figure 6 will be executed every 1 / 30th of a second. Alternatively, if the processing performance of the control device 102 is low, if power saving is a consideration, or if the communication environment is unstable, the frequency at which the flowchart in Figure 6 is executed may be reduced. Furthermore, the information or data generated in each processing step of the flowchart in Figure 6 executed at the previous timing (e.g., one frame earlier) is assumed to be stored in the RAM 312.
[0050] In S601, the communication control unit 416 receives the location information of the AR glasses 101 transmitted by the control device 102 in S502.
[0051] In S602, the area information acquisition unit 417 acquires information about a display area that has been pre-set in the real world.
[0052] Figure 7 shows an example of a display area in this embodiment. For example, suppose that four display areas, display area A, display area B, display area C, and display area D, are set in the real world. In Figure 7, region 701 represents the region of display area A, region 702 represents the region of display area B, region 703 represents the region of display area C, and region 704 represents the region of display area D. As shown in Figure 7, the display areas are set so that display areas B, display area C, and display area D are contained within display area A. In addition, the display areas are set so that parts of display area B and display area D overlap each other. In this way, the display areas may be set so that one display area overlaps with another. In S602, as information about the display areas, information about the position of each set display area in three-dimensional space is obtained.
[0053] In the flowchart explanation in Figure 6, for the sake of simplicity, each display area is assumed to be represented by a circle, and the coordinates of the circle's center and its radius are assumed to be obtained as the position information for each display area. Furthermore, the horizontal coordinates are assumed to be the X and Y coordinates, and it is assumed that each display area is set on the XY plane. For this reason, the coordinates shown below will only represent the X and Y coordinates in the explanation of the process. The shape of the display area is not limited to a circle; it can be a rectangle or any other shape. In that case, it is sufficient to obtain information that allows the coordinates of the boundary line of the display area to be determined as position information.
[0054] In S603, the entry / exit detection unit 412 determines a display area that includes the position of the AR glasses 101 from among the display areas set in the real world. For example, it compares the position information of the AR glasses 101 acquired in S601 with the position information indicating the area of each display area acquired in S602 to determine the display area in which the AR glasses 101 is located.
[0055] For example, let i represent the index of the display area, and the center coordinates (x) of the i-th display area i. i ,y i ) and the position coordinates (x AR ,y AR ) Distance d between two points i We can find this from Equation 1.
[0056]
number
[0057] Assume that each display area is circular in shape. In this case, as shown in Equation 2, the distance d between the center of the i-th display area i and the position of the AR glasses 101 is as follows: i However, the radius r of the i-th display area i i If the following conditions are met, it can be determined that the position of the AR glasses 101 is included inside the i-th display area i.
[0058]
number
[0059] For example, suppose that position 705 in Figure 7 is the position indicated by the position information of the AR glasses 101 acquired in S601. In this case, in S603, it is determined that the position of the AR glasses 101 is included inside the two display areas A and C.
[0060] In S604, the entry / exit detection unit 412 determines whether the AR glasses 101 have entered or exited any of the display areas. The entry / exit detection unit 412 stores the information of the display area including the position of the AR glasses 101, which was determined in S603 in the flowchart of Figure 6, which was executed at the previous timing. The entry / exit detection unit 412 then compares the display area including the position of the AR glasses 101 at the previous timing with the display area including the position of the AR glasses 101 determined at the current timing. Based on the comparison, the entry / exit detection unit 412 determines whether there has been a change in the display area including the position of the AR glasses 101. If there has been a change, it is determined that there has been an entry or exit to the display area (S604 is YES), and the process proceeds to S605; otherwise (S604 is NO), the process proceeds to S609.
[0061] In S605, the area dependency relationship acquisition unit 411 acquires information indicating the dependency relationships of the display areas.
[0062] Figure 8 shows an example of a dependency relationship between display areas. In Figure 8, A, B, C, and D represent display areas. For example, A represents display area A. In this embodiment, with respect to display areas set in the real world, one display area can be set to be subordinate to another display area. In Figure 8, the display area indicated by the arrow starting from a certain display area is a display area one level higher in the hierarchy to which that display area belongs.
[0063] In the example shown in Figure 8(a), display area C is configured to be subordinate to display area B, which is one level above it. Similarly, display area B is configured to be subordinate to display area A, which is one level above it. In this case, display area C is subordinate to both display area B and display area A.
[0064] Display area A is configured to be independent of any other display area because it does not have a higher level hierarchy. Display area D is one level above display area C, but as shown in Figure 8(a), there is no arrow pointing from display area C to display area D, indicating that display area C is configured not to be dependent on display area D. In this way, even display areas one level above can be configured not to be dependent. Display area B and display area D are at the same level hierarchy and therefore do not have a dependent relationship with each other (this is called a non-dependent relationship). In this embodiment, the dependency relationship is represented in a tree-like hierarchical structure, but the way in which the dependency relationship is represented is not limited to the tree-like method shown in Figure 8.
[0065] In S606, the display target area determination unit 413 determines the display area including the position of the AR glasses 101 determined in S603, and the display area to which the display area including the position of the AR glasses 101 belongs, as the "display target area".
[0066] For example, as shown in Figure 7, if the AR glasses 101 are at position 705, then in S603, it is determined that the display area including position 705 of the AR glasses 101 is display area C and display area A. Also, let's assume that information on the dependency relationships of the display areas shown in Figure 8(a) is acquired in S605. In this case, information on display area B, to which display area C belongs, and display area A, to which display area B belongs, is acquired. The display target area determination unit 413 then determines that the three display areas determined in S603—display area C including position 705 of the AR glasses 101, display area B to which display area C belongs, and display area A to which display area B belongs—are the display target areas. Note that since display area A has already been acquired as a display area including position 705 of the AR glasses 101, it does not need to be acquired again as a display area to which a display area belongs.
[0067] In S607, the content acquisition unit 414 acquires the CG content associated with each of the display target areas determined in S606 from the content management unit 418.
[0068] In S608, the display image generation unit 415 generates a two-dimensional display image from the CG content acquired in S607. The CG content managed by the content management unit 418 is, for example, CG content of a three-dimensional solid shape. The display image generation unit 415 sets virtual cameras corresponding to the left and right eyes according to the position, orientation, and field of view of the AR glasses 101 in three-dimensional space, and generates a two-dimensional display image by performing rendering processing on the three-dimensional CG content. The display image generation unit 415 performs rendering processing based on known rendering processes such as the rasterization method. The display image generation unit 415 stores the generated display image in the RAM 312.
[0069] In S609, the communication control unit 416 outputs the display image generated in S608 to the control device 102. As a result, in S503, the control device 102 of the AR glasses 101 receives the display image to be superimposed and performs the process of superimposing the received display image onto the real world.
[0070] On the other hand, if it is determined in S604 that there is no entry or exit to the display area (S604 is NO) and the process proceeds to S609, the communication control unit 416 retrieves the display image generated by the generation process in the most recent S608 from the RAM 312. The communication control unit 416 then outputs the retrieved display image to the control device 102. The initial value of the display image is a black image.
[0071] Figure 9(a) shows an example of a display image generated in S608. Figure 9(a) is an example of a display image when three display target areas, display area A, display area B, and display area C, are determined in S606. Therefore, Figure 9(a) is a display image generated from the CG content associated with each of the display areas A, B, and C. Display image 910 includes content image 903 generated from the CG content associated with display area A. It also includes content image 904 generated from the CG content associated with display area B, and content image 905 generated from the CG content associated with display area C.
[0072] Figure 9(b) shows the real-world space 920 as seen by a user wearing AR glasses 101. In other words, Figure 9(b) shows the state as seen by a user wearing AR glasses 101 when no superimposing of the displayed image has been performed.
[0073] Figure 9(c) shows the state a user sees when the displayed image 910 is superimposed onto the user's field of view while they are looking at the real space 920. In Figure 9(c), in addition to the real-world object 906, content images 903-905 that are included in the displayed image but do not exist in the real world are superimposed.
[0074] Figure 9(d) shows an example of CG content associated with display area D, which is not a target display area. CG content associated with a display area that is not a target display area is not included in the displayed image and becomes hidden content 907.
[0075] An example of how to set the dependency relationships of display areas shown in Figure 8(a) is explained below. For example, there may be a case where there are multiple stores in a shopping mall and you want to display different AR content to the user by overlaying it for each store. In this case, first, set the display area for each store. For example, in the example in Figure 7, the area 701 of the entire shopping mall is set as display area A. The area 702 of the parent store of the clothing store of brand α is set as display area B. The area 703 of the child store of the clothing store of the same brand α is set as display area C. The area 704 of the restaurant is set as display area D. For customers who visit a child store of the same brand as the parent store, you may want to display information about the parent store as AR content to the user. In such a case, in this embodiment, display area C of the child store of the clothing store of brand α is set to be subordinate to display area B of the parent store of the clothing store of the same brand α. With this setting, when a user wearing AR glasses 101 is in display area C of a child store, the user can see not only the content associated with display area C, but also the content associated with display area B of the parent store to which display area C belongs. Furthermore, the user can also see the content associated with display area A of the entire shopping mall.
[0076] Here, as a comparative example, let's consider a case where content corresponding to display areas A, B, and C is associated and registered in display area C, and content corresponding to display areas A and B is associated and registered in display area B. Furthermore, content corresponding to display areas A and D is associated and registered in display area D. In this comparative example as well, as a result, when the AR glasses 101 are in display area C, content corresponding to display areas A, B, and C will be displayed, just as described above.
[0077] Therefore, let's consider creating a new display area Z in the first layer and displaying the content corresponding to display area Z on the AR glasses 101 located in display areas B, D, and C. In this case, the comparative example method would require re-registering the content associated with each of the display areas B, D, and C. Similarly, if we want to prevent the display of content corresponding to display area Z, we would need to re-register the content associated with display areas B, D, and C. As the number of display areas increases, the comparative example method becomes even more complicated to manage. On the other hand, according to the method of this embodiment described above, when a new display area is created, it is only necessary to register the dependency relationship of the display areas as shown in Figure 8, thus reducing the effort required of the administrator to manage the content displayed to the user compared to the comparative example.
[0078] As described above, according to this embodiment, it is possible to control the display of CG content on a display area basis based on the display area including the position of the AR glasses 101 and the dependency relationships between each display area. Therefore, according to this embodiment, in order to control the content to be superimposed on the AR glasses 101, it is only necessary to set the dependency relationships between each display area, thus simplifying the management of CG content for displaying CG content.
[0079] The AR glasses 101 may also be a video pass-through (video see-through) type display device. In this case, the AR glasses 101 has an imaging unit such as a camera that captures the real world corresponding to the user's field of view. The AR glasses 101 then displays an image in which the display image is superimposed on the real-time image captured by the imaging unit on an opaque display (display unit) that the user can see. Furthermore, the device that superimposes and displays the display image transmitted from the information processing server 103 onto the real world is not limited to AR glasses. For example, the method of this embodiment can also be applied when superimposing a display image corresponding to a display area on a head-mounted display capable of displaying Mixed Reality, a smartphone, etc. When displaying a display image on an image capturing the real world corresponding to the user's field of view, the control device 102 processes the display image received from the information processing server 103, superimposing it onto the image and displaying it on an opaque display that the user can see.
[0080] Regarding the dependency relationships of display areas, a display area may be configured to be dependent on multiple display areas at the level above it. Figure 8(b) shows an example where the third-level display area C is configured to be dependent on the second-level display areas B and D. In Figure 8(b), when the user is in display area C, the content associated with display areas A, B, C, and D will be displayed. Furthermore, there is no limit to the number of display areas for which dependency relationships can be registered. For example, as shown in Figure 8(c), display area C may be configured to be the only dependent display area. In Figure 8(c), the content of display area B will be displayed in display area B, and the content of display areas B and C will be displayed in display area C.
[0081] In this embodiment, as shown in Figure 4, the information processing server 103 generates the display image and the control device 102 displays the display image. However, this configuration is just one example and other configurations are possible. For example, the control device 102 may have all or part of the functions of the information processing server 103, and the control device 102 may perform all or part of the processing described as the processing of the information processing server 103. For example, the control device 102 may include a display image generation unit 415. In this case, the communication control unit 416 of the information processing server 103 transmits data of CG content associated with the display target area to the control device 102, and the communication control unit 402 in the control device 102 receives the transmitted CG content data. The control device 102 may then perform rendering processing based on the received CG content data to generate a display image.
[0082] Alternatively, the control device 102 may include a content acquisition unit 414 and a display image generation unit 415, so that the control device 102 holds the CG content data. In this case, the communication control unit 416 in the information processing server 103 transmits information about the display target area to the control device 102, and the communication control unit 402 in the control device 102 receives the information about the display target area. The control device 102 then acquires the CG content data associated with the display target area, and the control device 102 may perform rendering processing based on the CG content data to generate a display image.
[0083] <Second Embodiment> This embodiment describes a method for generating a display image when the AR glasses 101 are positioned in an overlapping area of multiple display areas that are not dependent on each other. This embodiment will be described focusing on the differences from the first embodiment. Unless otherwise specified, the configuration and processing are the same as in the second embodiment.
[0084] Figure 10 shows an example of the configuration of the functions of the information processing server 103 according to this embodiment. The functions of the information processing server 103 according to this embodiment are configured with the functions shown in Figure 4 plus the display mode acquisition unit 1011. Therefore, the same numbers are used for the same functions as in Figure 4, and their explanations are omitted in Figure 10.
[0085] The display mode acquisition unit 1011 acquires the type of display mode currently set. The user or the operator of the information processing server 103 shall pre-set the type of display mode.
[0086] Figure 11 is a flowchart illustrating the display image generation process of this embodiment, which is executed in the information processing server 103. The series of processes shown in Figure 11 are realized when a program stored in the ROM 313 or HDD 314 of the information processing server 103 is loaded into the RAM 312 and executed by the CPU 311. As a result, the CPU 311 functions as the various parts shown in Figure 11.
[0087] Figure 11 is a flowchart of this embodiment corresponding to the flowchart in Figure 6. Therefore, for steps in Figure 11 that are the same as those in Figure 6, the same step numbers are used in Figure 11, and detailed explanations are omitted.
[0088] In this embodiment, S1101 is executed before S601. In S1101, the display mode acquisition unit 1011 acquires the type of currently set display mode. There are two types of display modes: "all display mode" and "priority display mode," and one of them is pre-set. "All display mode" is a display mode in which all display areas, including the position of the AR glasses 101 and its subordinate display areas, are the display target areas, while "priority display mode" is a display mode in which the display target area is determined based on the display priority of the display areas.
[0089] In this embodiment, the process proceeds from S605 to S1102. In S1102, the display target area determination unit 413 determines the display area including the position of the AR glasses 101 determined in S603, and the display area to which the display area including the position of the AR glasses 101 belongs, as "candidate display target areas". The candidate display target areas are display areas that are candidates for display.
[0090] Figure 12 shows an example of a display area in this embodiment. For example, suppose that in the real world, six display areas A through F are set up. Display areas B and D overlap in part, and display areas C and E overlap in part. The areas where the display areas overlap are called overlapping areas.
[0091] Figure 13 shows an example of the dependency relationships of the display areas shown in Figure 12. Display area F is set to be dependent on display areas C, B, and A, which are in the higher hierarchy. Display areas C and E, which are in the third hierarchy, are not dependent on each other. Display areas B and D, which are in the second hierarchy, are also not dependent on each other.
[0092] For example, let's assume that position 1207, shown in Figure 12, is the position of the AR glasses 101. In this case, a total of five display areas are determined as "candidate display areas" in S1102: three display areas A, C, and E that include position 1207, and two display areas B and D to which any of these three display areas belong.
[0093] In S1103, the display target area determination unit 413 determines whether the AR glasses 101 are located in multiple display areas that are not dependent on each other. If the display target area determination unit 413 determines that the AR glasses 101 are located in multiple display areas that are not dependent on each other (S1103 is YES), it proceeds to S1104; otherwise (S1103 is NO), it skips S1104 and proceeds to S1105.
[0094] As shown in Figure 12, the position 1207 of the AR glasses 101 is included in the overlapping region where display area C and display area E overlap. Also, as shown in Figure 13, display area C is not dependent on display area E, and display area E is not dependent on display area C, so display area C and display area E are in an independent relationship with no dependency relationship. Therefore, if the AR glasses 101 is at position 1207 in Figure 12, it is determined that it is in multiple display areas that are not dependent on each other (S1103 is YES).
[0095] In S1104, the display target area determination unit 413 determines whether the currently set display mode is "all display mode" based on the type of display mode acquired in S1101.
[0096] If the display target area determination unit 413 determines that the current display mode is "all display mode" (S1104 is YES), it proceeds to S1105. In S1105, the display target area determination unit 413 determines the display target area in the same way as described in the first embodiment. Specifically, it determines all of the candidate display target areas determined in S1102 as display target areas. That is, if the AR glasses 101 are at position 1207 in Figure 12 and the current display mode is "all display mode", then five display areas, A, B, C, D, and E, are determined as display target areas.
[0097] On the other hand, if the display target area determination unit 413 determines that the current display mode is not the all-display mode (S1104 is NO), it proceeds to S1106. That is, if the current display mode is the "priority display mode", it proceeds to S1106. In S1106, the display target area determination unit 413 determines the display target area from the candidate display target areas determined in S1102 based on the priority of the display area.
[0098] Specifically, the display target area determination unit 413 selects multiple display areas that include the position of the AR glasses 101 and that are not dependent on each other as processing targets. Then, it determines the display area with the highest priority among the multiple display areas to be processed. The display target area determination unit 413 determines the display target area to be the determined high-priority display area and the display areas to which the high-priority display area is dependent. In other words, the display target area determination unit 413 determines the display target area to be the display area by assuming that the AR glasses 101 is not present in any display area other than the high-priority display area among the display areas that constitute the overlapping region.
[0099] For example, as shown in Figure 12, if the AR glasses 101 are at position 1207, the display areas including position 1207 are display area C, display area E, and display area A. Of these three display areas, the display areas that are not dependent on each other are display area C and display area E, as shown in Figure 13. Therefore, in S1106, the combination of display area C and display area E is selected as the processing target, and the display area with the highest priority among display area C and display area E is determined to be the display target area, while the other display areas are not determined to be the display target area.
[0100] In this embodiment, as shown in Figure 13, the information indicating the dependency relationship between display areas includes a pre-set display priority for each display area. For example, in Figure 13, display area C, included in the third layer, has a priority of 1, while display area E has a priority of 2. In this embodiment, a lower priority value is determined to indicate a higher priority. Therefore, comparing the priorities of display area C and display area E, display area C is set as a display area with a higher display priority.
[0101] Therefore, in S1106, if the display area to be processed is a combination of display area C and display area E, display area C is determined as the display target area, and display area E is not determined as the display target area. Furthermore, the display area to which display area C, which has been determined as the display target area, belongs is also determined as the display target area. Therefore, display area B and display area A, to which display area C belongs, are also determined as the display target area. To summarize, in Figure 12, if the AR glasses 101 are at position 1207 and the current display mode is "priority display mode", then display areas A, B, and C are determined as the display target areas.
[0102] Figure 14(a) shows the view of a user wearing AR glasses 101 when the current display mode is "All Display Mode". As mentioned above, if the AR glasses 101 are at position 1207 in Figure 12, and the dependency relationship of the display areas is as shown in Figure 13, and the display mode is "All Display Mode", then five display areas A, B, C, D, and E are determined as the display target areas. Therefore, in S607, CG content associated with each of the display areas A, B, C, D, and E is acquired, and in S608, a display image is generated using the acquired CG content. Thus, in the field of view of the user wearing AR glasses 101, in addition to the real-world object 1408, content images 1403 to 1407 that do not exist in the real world are superimposed. Content image 1403 is an image generated from the CG content associated with display area A. Similarly, content image 1404 is an image generated from CG content associated with display area B, content image 1405 is an image generated from content content associated with display area C, content image 1406 is an image generated from content content associated with display area D, and content image 1407 is an image generated from content content associated with display area E. Figure 14(b) shows an example of CG content 1409 associated with display area F, which was not determined to be a display target area. In this embodiment as well, CG content associated with display areas that were not determined to be display target areas is not included in the displayed image and is not in the user's field of view.
[0103] Figure 15(a) shows the view of a user wearing AR glasses 101 when the current display mode is "priority display mode". As mentioned above, if the AR glasses 101 are at position 1207 in Figure 12, and the dependency relationship of the display areas is as shown in Figure 13, and the display mode is "priority display mode", then three display areas A, B, and C are determined as the display target areas. Therefore, in S607, CG content associated with each of the display areas A, B, and C is acquired, and in S608, a display image is generated using the acquired CG content. Thus, in the field of view of the user wearing AR glasses 101, in addition to the real-world object 1408, the aforementioned content images 1403-1405, which do not exist in the real world, are superimposed. Figure 15(b) shows an example of CG content associated with display areas D, E, and F, which were not determined to be display target areas.
[0104] This section explains a specific example of how to set the priority of display areas. For example, in Figure 12, suppose the area 1202 of the parent store of brand α's clothing store is set as display area B, and the area 1203 of the child store of brand α's clothing store is set as display area C. Also, suppose the area 1204 of one restaurant is set as display area D, and the area 1205 of another restaurant is set as display area E. In "priority display mode," among the content images 1403 to 1407 that were displayed in "all display mode," content images 1406 and 1407 corresponding to lower priority display areas are generated so that the user cannot see them. Therefore, as shown in Figure 15(a), content images 1403 to 1405 associated with higher priority display areas can be displayed in a way that makes them easier for the user to notice. For example, if you want to hold a fair at a clothing store in a shopping mall and guide customers to the clothing store, you should set the priority of the clothing store's display areas B and C higher than the display areas D and E of the restaurants. By setting priorities in this way, content images such as advertisements for clothing stores associated with the display area of the clothing store can be displayed in a way that makes them more likely to catch the user's eye.
[0105] As described above, according to this embodiment, even when the AR glasses 101 are positioned in an overlapping area of multiple display areas that are not dependent on each other, it becomes possible to appropriately control the display of CG content for multiple display areas according to the display mode.
[0106] Furthermore, overlapping areas may be set between display areas of different hierarchical levels. In this case, if the AR glasses 101 are located in such overlapping area and the display mode is set to priority display mode, the display control may be set to prioritize the display area located in the higher hierarchical level.
[0107] <Third Embodiment> In the second embodiment, a method for controlling the display based on priority was described when the AR glasses 101 are located in an overlapping area of multiple display areas that are not dependent on each other. In this embodiment, a method for controlling the display based on the direction of movement of the AR glasses 101 will be described. This embodiment will mainly describe the differences from the first or second embodiment. Unless otherwise specified, the configuration and processing are the same as in the first or second embodiment.
[0108] Figure 16 shows an example of the configuration of the functions of the information processing server 103 according to this embodiment. The functions of the information processing server 103 according to this embodiment are configured with the functions shown in Figure 10 plus a movement direction specification unit 1611. Therefore, the same numbers are used for the same functional units as in Figure 10, and their descriptions are omitted.
[0109] The movement direction determination unit 1611 determines the direction of movement based on the amount of change in the position information of the AR glasses 101.
[0110] Figure 17 is a flowchart illustrating the display image generation process of this embodiment, which is executed in the information processing server 103. The series of processes shown in Figure 17 are realized when a program stored in the ROM 313 or HDD 314 of the information processing server 103 is loaded into the RAM 312 and executed by the CPU 311. As a result, the CPU 311 functions as the various parts shown in Figure 16.
[0111] Figure 17 is a flowchart of this embodiment corresponding to the flowchart in Figure 11. Therefore, for steps in Figure 17 that are the same as those in Figure 11, the same step numbers as in Figure 5 are used in Figure 17, and detailed explanations are omitted.
[0112] In S1701, the display mode acquisition unit 1011 acquires the type of currently set display mode. In this embodiment, it is assumed that there are two types of display modes: "all display mode" and "not all display mode". If a priority is set for the display area, there may be an additional "priority display mode" among the display mode types.
[0113] In S1104, if the display target area determination unit 413 determines that the current display mode is "not full display mode" (S1104 is NO), in this embodiment, the processing in S1702 to S1703 is performed. If the current display mode is "full display mode" (S1104 is YES), the same processing as in the second embodiment is performed.
[0114] In S1702, the movement direction determination unit 1611 determines the movement direction of the AR glasses 101 based on the amount of time change in the position of the AR glasses 101.
[0115] Figure 18 illustrates how to determine the direction of movement of the AR glasses 101. In Figures 18(a) and 18(b), position 1805 indicates the current position of the AR glasses 101. Positions 1806 and 1807 indicate the position information of the AR glasses 101 acquired at the previous timing. The information for position 1806 or position 1807 is stored in the RAM 312 as described above. The temporal change in the position of the AR glasses is used to determine the direction from which the AR glasses 101 moved.
[0116] In S1703, the display target area determination unit 413 determines the display target area based on the movement direction of the AR glasses 101 acquired in S1702.
[0117] Specifically, the display target area determination unit 413 selects multiple display areas, including the position of the AR glasses 101, that are not dependent on each other, as the display target area for processing. Then, it processes the multiple display areas to be processed so that the display area in the direction from which the AR glasses 101 has moved is not included as the display target area. In other words, it determines the display area in the direction from which the AR glasses 101 is moving to be the display target area from among the multiple display areas to be processed.
[0118] For example, as shown in Figures 18(a) and 18(b), if the AR glasses 101 are at position 1805, the display areas including position 1805 are display area A, display area B, and display area D. Furthermore, the dependency relationships of the set display areas are as shown in Figure 8(a). In this case, of these three display areas A, B, and D, the display areas that are not dependent are display area B and display area D. Therefore, in S1703, the combination of display area B and display area D is selected as the processing target. Then, among display area B and display area D, the display area that the AR glasses 101 is facing (the display area on the side of the direction of movement of the AR glasses 101) is determined as the display target area. That is, the display area in the direction from which the AR glasses 101 came (the display area on the side opposite to the direction of movement of the AR glasses 101) is not determined as the display target area.
[0119] In Figure 18(a), the position of the AR glasses at the previous timing was position 1806, so the direction of movement is determined to be the lower left direction. Therefore, information is obtained that the AR glasses 101 have moved from display area B to position 1805. Therefore, it is decided that display area B will not be a target area for display. Thus, in the case of display area D and display area B as the combination of display areas to be processed, display area D is determined to be a target area for display, and display area B is not determined to be a target area for display. Furthermore, the display area to which display area D, which has been determined to be a target area for display, belongs is also determined to be a target area for display. Thus, in Figure 18(a), display areas A and D are determined to be target areas for display.
[0120] In Figure 18(b), the position of the AR glasses at the previous timing was position 1807, so the direction of movement is determined to be towards the upper right. Therefore, information is obtained that the AR glasses 101 have moved from display area D to position 1805. For this reason, if the combination of display areas to be processed is display area D and display area B, display area B is determined to be the display target area, and display area D is not determined to be the display target area. Furthermore, the display area to which display area B, which has been determined to be the display target area, belongs is also determined to be the display target area. Thus, in Figure 18(b), display areas A and B are determined to be the display target areas.
[0121] As described above, according to this embodiment, even when the AR glasses 101 are located in an overlapping area of multiple display areas that are not dependent on each other, it becomes possible to appropriately control the display of CG content for multiple display areas based on the direction of movement of the AR glasses 101.
[0122] <Other Embodiments> This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the program may be recorded on a recording medium readable by a computer and provided.
[0123] The above-described embodiments include the following configurations.
[0124] (Composition 1) An information processing device that outputs a display image to a display device that processes the display image so that the user can see the real world with the display image superimposed on it, A management means for managing content associated with each of multiple areas in the real world, and the dependency relationships between the areas that constitute the multiple areas, A generation means for generating a display image that includes images corresponding to content associated with the area including the location of the display device among the plurality of areas, and content associated with the area to which the area including the location of the display device belongs. An information processing device characterized by having the following features. (Configuration 2) The system further includes output means for outputting the generated display image to the display device. The information processing device according to configuration 1, characterized by the above. (Composition 3) The output means is The display image is transmitted to the display device at predetermined intervals. If the display device does not enter or exit any of the multiple areas, the generation means outputs the most recently generated display image to the display device. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) When the display device enters or exits any of the multiple areas, the system further includes a determination means for determining the area containing the location of the display device among the multiple areas, and the area to which the area containing the location of the display device belongs, as the display target area. When the determination means determines the display target area, the generation means generates the display image which includes an image corresponding to the content associated with the display target area. An information processing device according to any one of configurations 1 to 3. (Composition 5) The system further includes detection means for detecting when there is a change in the area including the position of the display device, that the display device has entered or left one of the plurality of areas. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) Priorities are assigned to the aforementioned multiple areas. The aforementioned determination means is If the display device is located in an area where two or more of the aforementioned areas that are not dependent on each other overlap, the area to be displayed is determined by excluding the area with the highest priority among the two or more areas from which the display device is located. The information processing apparatus according to configuration 4 or 5, characterized by the above. (Composition 7) There are two types of display modes: Mode 1 and Mode 2. The aforementioned determination means is When the display device is located in the overlapping region, If the set display mode is the first mode described above, the display target area is determined based on priority, If the set display mode is the second mode, the display target area is determined such that the location of the display device is included in all of the two or more areas. The information processing device according to configuration 6, characterized by the features described therein. (Composition 8) The display device further has a means for identifying the direction in which it is moving. The aforementioned determination means is If the display device is located in an area where two or more of the aforementioned areas, which are not dependent on each other, overlap, the area to be displayed is determined by excluding the location of the display device from the area among the two or more areas that is not in the direction in which the display device is moving. The information processing apparatus according to configuration 4 or 5, characterized by the above. (Composition 9) There are two types of display modes: a second mode and a third mode. The generating means is When the display device is located in the overlapping region, If the set display mode is the third mode described above, the display target area is determined based on the direction in which the display device is moving. If the set display mode is the second mode described above, the display area is determined such that the location of the display device is included in all of the two or more areas. The information processing apparatus according to configuration 8, characterized by the above. (Composition 10) The aforementioned dependency relationship is a dependency relationship expressed in a hierarchical structure. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above. (Composition 11) A receiving means for receiving the display image from the information processing device described in any one of configurations 1 to 10, A display control means that processes the received display image to display the real world so that the user can see it superimposed on it, A display device characterized by having the following features. (Composition 12) The display image is superimposed on the real world, and the display image is configured to come into the user's eye when combined with light from the real world, so that the user can see the real world. A display device according to configuration 11, characterized by the features described above. (Composition 13) Imaging means, It has a display means, The display control means superimposes the display image onto the captured image obtained by the imaging means capturing the real world and displays it on the display means. A display device according to configuration 11, characterized by the features described above. (Composition 14) The system further comprises a transmission means for transmitting the location information of the display device to the information processing device described in configuration 1. A display device according to any one of the configurations 11 to 13, characterized by the features described above. (Composition 15) A control device for a display device, A receiving means for receiving the display image from the information processing device described in any one of configurations 1 to 10, A display control means that processes the display of the display device so that the user can see the real world on which the display image is superimposed, A control device characterized by having the following features. (Composition 16) A method for generating a display image to be output to a display device that processes the display image so that the user can see the real world superimposed on the display image, A step to obtain the dependency relationships between areas that constitute multiple areas in the real world, A generation step of generating a display image that includes images corresponding to content associated with the area including the location of the display device among the plurality of areas, and content associated with the area to which the area including the location of the display device belongs. An image generation method characterized by having the following features. (Composition 17) A program that causes a computer to execute each of the means of the information processing device described in any one of the configurations 1 to 10. [Explanation of Symbols]
[0125] 101 AR Glasses 103 Information Processing Server 418 Content Management Department 415 Display Image Generation Unit
Claims
1. An information processing device that outputs a display image to a display device that processes the display image so that the user can see the real world in which the display image is superimposed, A management means for managing content associated with each of multiple areas in the real world, and the dependency relationships between the areas that constitute the multiple areas, A generation means for generating a display image that includes images corresponding to content associated with the area including the location of the display device among the plurality of areas, and content associated with the area to which the area including the location of the display device belongs. An information processing device characterized by having the following features.
2. The system further includes output means for outputting the generated display image to the display device. The information processing apparatus according to feature 1.
3. The output means is The display image is transmitted to the display device at predetermined intervals. If the display device does not enter or exit any of the multiple areas, the generation means outputs the most recently generated display image to the display device. The information processing apparatus according to feature 2.
4. When the display device enters or exits any of the multiple areas, the system further includes a determination means for determining the area containing the location of the display device among the multiple areas, and the area to which the area containing the location of the display device belongs, as the display target area. When the determination means determines the display target area, the generation means generates the display image which includes an image corresponding to the content associated with the display target area. The information processing apparatus according to feature 1.
5. The system further includes detection means for detecting when there is a change in the area including the position of the display device, that the display device has entered or left one of the plurality of areas. The information processing apparatus according to feature 4.
6. Priorities are assigned to the aforementioned multiple areas. The aforementioned determination means is If the display device is located in an area where two or more areas that are not subordinate to each other overlap, the area to be displayed is determined by excluding the area with the highest priority among the two or more areas from which the display device is located. The information processing apparatus according to feature 4.
7. There are two types of display modes: a first mode and a second mode. The aforementioned determination means is When the display device is located in the overlapping region, If the set display mode is the first mode described above, the display target area is determined based on priority, If the set display mode is the second mode, the display target area is determined such that the location of the display device is included in all of the two or more areas. The information processing apparatus according to feature 6.
8. The display device further has a means for identifying the direction in which it is moving. The aforementioned determination means is If the display device is located in an area where two or more of the aforementioned areas that are not dependent on each other overlap, the area of the two or more areas that is not in the direction in which the display device is moving is excluded from the display area, and the display target area is determined accordingly. The information processing apparatus according to feature 4.
9. There are two types of display modes: a second mode and a third mode. The generating means is When the display device is located in the overlapping region, If the set display mode is the third mode described above, the display target area is determined based on the direction in which the display device is moving. If the set display mode is the second mode, the display target area is determined such that the location of the display device is included in all of the two or more areas. The information processing apparatus according to feature 8.
10. The aforementioned dependency relationship is a dependency relationship expressed in a hierarchical structure. The information processing apparatus according to feature 1.
11. Receiving means for receiving the display image from the information processing device described in claim 1, A display control means that processes the received display image to display the real world so that the user can see it superimposed on it, A display device characterized by having the following features.
12. The display image is superimposed on the real world, and the display image is configured to come into contact with the user's eyes in combination with light from the real world, so that the user can see the real world. The display device according to feature 11.
13. Imaging means, It has a display means, The display control means superimposes the display image onto the captured image obtained by the imaging means capturing the real world and displays it on the display means. The display device according to feature 11.
14. The device further comprises a transmission means for transmitting the position information of the display device to the information processing device described in claim 1. The display device according to feature 11.
15. A control device for a display device, Receiving means for receiving the display image from the information processing device described in claim 1, A display control means that processes the display of the display device so that the user can see the real world on which the display image is superimposed, A control device characterized by having the following features.
16. A method for generating a display image to be output to a display device that processes the display image so that the user can see the real world superimposed on the display image, A step to obtain the dependency relationships between areas that constitute multiple areas in the real world, A generation step of generating a display image that includes images corresponding to content associated with the area including the location of the display device among the plurality of areas, and content associated with the area to which the area including the location of the display device belongs. An image generation method characterized by having the following features.
17. A program for causing a computer to execute each of the means of the information processing apparatus described in any one of claims 1 to 10.