Apparatus and method for displaying virtual object

Abstract

To provide a device for displaying a virtual object capable of enhancing the presence of a user even in a situation where the virtual object cannot be visually recognized, and a display method thereof.SOLUTION: A head-mounted display (HMD) 1 that displays a virtual object includes a map information processing unit that extracts a first map element corresponding to position information and a second map element corresponding to a predetermined pattern from map data, a virtual object processing unit that arranges the virtual object on a real object in a real space corresponding to the first map element, and an auxiliary information processing unit that generates a map element object corresponding to the second map element. An information processing device includes a display processing unit that makes a drawing method different between a portion located in front of a real object and a portion located behind the real object, and a screen that superimposes and displays a virtual object and a map element object processed by the display processing unit on a real space.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a device for displaying a virtual object, such as a head-mounted display, and a display method thereof. [Background technology]

[0002] Mixed reality (MR) technology, which superimposes virtual objects created using CG (Computer Graphics) onto real space, is widely used in games, sports, remote medical care, maintenance work, and more.

[0003] Devices that display virtual objects include, for example, information processing devices such as HMDs (Head Mounted Displays), HUDs (Head Up Displays) mounted on vehicles and aircraft, car navigation systems, and smartphones.

[0004] In MR technology, for example, in the case of an HMD, virtual objects are drawn in response to the movement of the HMD and are superimposed on an image of real space seen through the display unit, and are displayed on the display unit as an image of virtual space. Alternatively, in an HMD, an image of real space captured by a camera and virtual objects are superimposed on a non-transparent reflective display unit.

[0005] Background art in this technical field is Patent Document 1. Patent Document 1 discloses that the user of an HMD is a runner, and a virtual object representing a virtual runner is displayed on the HMD to enhance the sense of realism of the running experience and allow the user to easily grasp information about the running. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-67277 Summary of the Invention [Problem to be solved by the invention]

[0007] MR technology compares the distance from the user to a real object in real space with that of a virtual object, and performs occlusion processing to hide the parts of the virtual object that are farther away than the real object, achieving a three-dimensional view. Therefore, if the entire virtual object is hidden, the user cannot see the virtual object.

[0008] In Patent Document 1, if the user continues to be unable to see the virtual object, the user will be unable to recognize the virtual runner for a long period of time, making it difficult to achieve the goal of enhancing the sense of realism. Thus, Patent Document 1 does not consider how to deal with the situation where the virtual object becomes unrecognizable due to occlusion processing with a real object.

[0009] The present invention has been made in consideration of the above, and its object is to provide a device and a display method for displaying virtual objects that can enhance the user's sense of realism even in situations where the virtual objects cannot be seen. [Means for solving the problem]

[0010] As one example, the present invention is a method for displaying virtual objects, comprising a map information processing step of extracting from map data a first map element corresponding to location information and a second map element corresponding to a predetermined pattern; a virtual object processing step of placing a virtual object on a real object in real space corresponding to the first map element; an auxiliary information processing step of generating a map element object corresponding to the second map element; a display processing step of using different drawing methods for portions of the virtual object and the map element object that are located in front of and behind the real object; and a display step of superimposing and displaying the virtual object and map element object processed in the display processing step on real space. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a device and a display method for displaying a virtual object that can deal with situations where the virtual object cannot be seen and can enhance the sense of realism of the user. [Brief explanation of the drawings]

[0012] [Figure 1A] 1 is an external configuration diagram of an HMD according to an embodiment. [Figure 1B] FIG. 10 is an external configuration diagram of another HMD in the embodiment. [Figure 2] FIG. 2 is a functional block diagram of an HMD according to an embodiment. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of an HMD according to an embodiment. [Figure 4] 10 is a flowchart of MR processing in the embodiment. [Figure 5] 10 is a flowchart of a process for generating a virtual object in the embodiment. [Figure 6] 10 is a flowchart of a process for arranging a virtual object in the embodiment. [Figure 7] 10 is a flowchart of a process for generating a course object in an embodiment. [Figure 8] 10 is a display example of a visible virtual object in the embodiment. [Figure 9] 10 is a diagram illustrating an example of display of a virtual object when the virtual object is not visible in the embodiment. [Figure 10] 10 is an example of placing a virtual object at a specific position in the embodiment. [Figure 11] 10 is a display example in which a virtual object is located behind the user in the embodiment. [Figure 12] 10 is a display example in the case where a virtual object is out of the visible range in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that in this embodiment, an HMD will be described as an example of a device for displaying virtual objects. [Example]

[0014] Fig. 1A is an external configuration diagram of the HMD in this embodiment. In Fig. 1A, 1 is an HMD, 10 is a camera, 11 is a distance measurement sensor, 12a and 12b are a pair of left and right projection units (projectors), 13 is a semi-transparent screen, 14 is a speaker, 15 is a microphone, 16 is a housing, 17 is a support unit, and 18 is a control unit.

[0015] A user of the HMD 1 wears the HMD 1 on their face using the housing 16 and the support part 17. The camera 10 captures an image of the real space in front of the HMD 1, and the distance measurement sensor 11 measures the distance between the HMD 1 and a real object in the real space captured by the camera.

[0016] The projection units 12a and 12b and the screen 13 constitute the display unit of the HMD 1. The projection units 12a and 12b project an image of a virtual object to be seen by the left eye and an image of a virtual object to be seen by the right eye, respectively, onto the screen 13, and display the virtual objects, which are the projected images, in three dimensions as if they were located at a predetermined distance in real space.

[0017] In this embodiment, the HMD is described as an optical see-through type in which the HMD user sees an image of the real space in front of them through the screen 13, but it may also be a video see-through type in which an image of the real space captured by a camera is projected onto the screen 13 and viewed.

[0018] In the HMD display, occlusion processing is performed based on the distance between real and virtual objects. Occlusion processing is a process of processing the rendering data of virtual objects so that when part of a real object is in front of part of a virtual object, part of the virtual object appears to be hidden by that part of the real object, thereby displaying an image in an MR space with depth.

[0019] The control unit 18 captures images of real space captured by the camera 10 and supplies them to an internal memory and CPU. The HMD 1 also has built-in sensors such as a GPS, gyro, orientation sensor, and acceleration sensor, and the control unit 18 detects the position and movement of the HMD based on information from these sensors. The control unit 18 also creates images to be projected by the projection units 12a and 12b and sounds to be output to the speaker 14. The control unit 18, camera 10, distance measurement sensor 11, speaker 14, and microphone 15 are arranged in a housing 16. Note that their locations are not limited to those shown in FIG. 1A.

[0020] Fig. 1B is an external configuration diagram of another HMD in this embodiment. In Fig. 1B, the same functions as in Fig. 1A are assigned the same reference numerals, and their explanations will be omitted. Fig. 1B differs from Fig. 1A in that control unit 18 is divided into 18a and 18b.

[0021] 1B, control unit 18a and control unit 18b are connected via a wired or wireless interface. Control unit 18b is a general-purpose information terminal such as a smartphone or a smartwatch. With the configuration of HMD2 shown in FIG. 1B, part of control unit 18 shown in FIG. 1A can be provided as control unit 18a, which has the advantage of allowing for a smaller and lighter HMD2.

[0022] Fig. 2 is a functional block diagram of the HMD in this embodiment. Fig. 2 shows the case of the HMD 1 in Fig. 1A, and particularly shows the details of the functional block diagram of the control unit 18. Note that the same blocks as in Fig. 1A are given the same reference numerals, and their explanations will be omitted. Also, the projection units 12A and 12B in Fig. 1A are collectively referred to as the projection unit 12, and the microphone, speaker, etc. are omitted.

[0023] In FIG. 2, the control unit 18 has an image recognition processing unit 20, a communication unit 21, a map information processing unit 22, a virtual object processing unit 23, a display processing unit 24, a position detection processing unit 25, an auxiliary information processing unit 26, and an overall control unit 27.

[0024] The position detection processing unit 25 includes a GPS, a direction sensor, a gyro sensor, etc., and detects the position and orientation of the HMD. Based on the information detected by the position detection processing unit 25, the overall control unit 27 obtains the distance between the HMD (=user) and the virtual object (=competitor) calculated by the virtual object processing unit 23. Then, based on this information, the overall control unit 27 specifies a position and range to a map data server (not shown) via the communication unit 21, and requests downloading. The map data downloaded by the communication unit 21 is input to the map information processing unit 22.

[0025] The map information processing unit 22 extracts map elements such as roads that are set in advance as a driving course from the map data, and outputs the extracted information to the virtual object processing unit 23. The map information processing unit 22 holds course data of the user's planned driving course, or obtains this data via the communication unit 21, and uses it to extract map elements from the map data.

[0026] The image recognition processing unit 20 receives the camera image from the camera 10 and the distance data from the distance measurement sensor 11, recognizes real objects such as roads and buildings from the real space captured by the camera image, and assigns distance data to the feature points of the real objects.

[0027] The virtual object processing unit 23 calculates the position of the competitor based on the competitor's running pace information, and further generates image data of the virtual object. The image data of the virtual object may be obtained from an external server via the communication unit 21. The virtual object processing unit 23 further obtains the user's current position from the position detection processing unit 25, and places the virtual object according to extracted information such as the running course. The position of the virtual object is then sent to the position detection processing unit 25 to determine the range of map data to be downloaded. The download range of map data changes from moment to moment, but the download data volume can be kept small by updating the difference from the range that has already been downloaded.

[0028] The auxiliary information processing unit 26 performs processing such as generating course objects of auxiliary information that conform to the course data from extracted information such as the driving course, and the display processing unit 24 inputs the course objects from the auxiliary information processing unit 26, the virtual objects from the virtual object processing unit 23, and the real objects from the image recognition processing unit 20, performs occlusion processing between the virtual objects and course objects and the real objects, and sends the virtual object images and course object images to the projection unit 12 to display them on the screen 13.

[0029] Fig. 3 is a block diagram of the hardware configuration of the HMD in this embodiment, showing the case of HMD 1 in Fig. 1A. In Fig. 3, the same blocks as in Fig. 1A are given the same reference numerals, and their explanations will be omitted. Note that control unit 18 is shown divided into control units 18a and 18b shown in Fig. 1B.

[0030] 3, the control unit 18a includes a sensor group 28 including a GPS, a direction sensor, a gyro sensor, etc., and an interface unit 29, and the control unit 18b includes a communication unit 30, a CPU 31, a RAM 32, a Flash ROM (FROM) 33, and an interface unit 36.

[0031] The communication unit 30 of the control unit 18b selects an appropriate process from several communication processes, such as mobile communications (e.g., 4G, 5G), wireless LAN, etc., connects the HMD to the network, and downloads map data, etc., from an external server. Furthermore, the FROM 33 includes a basic program 34 and an MR processing program 35 as processing programs. These processing programs can be expanded in the RAM 32 and processed by the CPU 31 as software, thereby realizing the various functions shown in FIG. 2. Furthermore, the FROM 33 stores data necessary for executing the processing programs. The FROM 33 may be a single memory medium as shown, or may be composed of multiple memory media. It may also be a non-volatile memory medium other than a flash ROM.

[0032] In FIG. 3, in the case of the configuration of the HMD 1 having the control unit 18 in which the control units 18a and 18b are integrated as shown in FIG. 1A, the interface units 29 and 36 may be omitted.

[0033] 1B, in the case where control unit 18 is divided into control units 18a and 18b, control unit 18b is separated from the HMD in FIG. 3, and control units 18a and 18b are connected by interface units 29 and 36. In this case, interface units 29 and 36 may be wired, such as USB (registered trademark), or wireless, such as wireless LAN or Bluetooth (registered trademark). As described above, in the case where control unit 18 is divided into control units 18a and 18b, the HMD only needs to be provided with control unit 18a, which is a part of control unit 18, and therefore the HMD can be made smaller and lighter.

[0034] Fig. 4 is a flowchart of the MR processing in this embodiment. In Fig. 4, the process starts at step S10, and step S11 is the position detection process of the position detection processing unit 25 described in Fig. 2. In step S11, the current position of the HMD is detected using data from the sensor group 28, and a request to download map data is output along with the positions of virtual objects (described later). In addition, in step S12, the current position of the HMD and the positions of virtual objects are stored as a driving record at regular intervals.

[0035] Steps S13 to S15 are map information processing steps performed by the map information processing unit 22 described in Fig. 2. Map data is downloaded in S13, course data is read in S15, and the read course data is referenced in S14 to extract roads and the like set as the driving course as map elements.

[0036] Step S16 is a step of camera photography and distance measurement in the camera processing unit, where the camera image and distance data are taken in.

[0037] Steps S17 and S18 are image recognition processes performed by the image recognition processing unit 20 described in Fig. 2. In S17, real objects such as roads and buildings are recognized from the camera image, which is an image of real space, and in S18, distance data is associated with feature points of the recognized real objects.

[0038] Steps S20 to S23 are virtual object processing by the virtual object processing unit 23 described in FIG. 2. Stored running pace data is read in S21, the running distance of the virtual object is calculated in S20, and an image of the virtual object (= competitor) is generated in S22. The size of the image of the virtual object changes depending on the direct viewing distance from the HMD, and the orientation changes depending on the direction of the HMD. Furthermore, in S23, the virtual object is placed on the road of the extracted running course. Furthermore, the placement position of the virtual object is sent to position detection processing S11, thereby determining the range of map data to be downloaded.

[0039] Step S24 is auxiliary information processing by the auxiliary information processing unit 26 described in FIG. 2, which generates a course object (= auxiliary information object) corresponding to the extracted driving course. The course object is an object that indicates the road along the driving course, and may be a three-dimensional object that reflects distance data. Furthermore, distance information objects that indicate numerical data on distance information may be added as milestones as auxiliary information objects. Furthermore, if a competitor is outside the user's field of view, a pseudo course object may be generated instead of the course object. The pseudo course object may reflect only the sense of distance from the competitor. Note that these course objects, distance information objects, pseudo course objects, etc. are objects related to the map, and are therefore also referred to as map element objects.

[0040] Steps S25 to S27 are the display processing of the display processing unit 24 described in Fig. 2. In S25, occlusion processing between real objects and virtual objects (including course objects) is performed. In the occlusion processing, the real objects and virtual objects in the real space are compared in terms of distance from the user, and a distinction is made between a portion of the virtual object that is closer than the real object and a portion of the virtual object that is farther away than the real object.

[0041] When viewed from the user, the portion of the virtual object that is farther away than the real object is hidden by the real object and becomes invisible. However, if the virtual object remains invisible, the user will not be able to recognize the competitor, and the motivation of the user to drive while checking the competitor will not be improved. For this reason, in S26, the portion of the virtual object that is hidden by the real object and cannot be seen is drawn differently from the visible portion of the virtual object. The different drawing is achieved, for example, by using a different color scheme.

[0042] In step S27, the virtual object and auxiliary information object are output and projected onto the display unit of the HMD. Then, in step S28, the end of the program is confirmed. If not (No), the process returns to immediately after S10. If it is (Yes), the process ends in S29.

[0043] Fig. 5 is a flowchart of the virtual object generation process (S22) in Fig. 4. In Fig. 5, it is determined in S50 whether the competitor is ahead of the user, and if so (YES), a virtual object is generated in real space in S51, and if not (NO), a rearview mirror object is generated and the virtual object is placed inside the rearview mirror object in S52.

[0044] FIG. 6 is a flowchart of the virtual object placement process (S23) in FIG. 4. In FIG. 6, the road width is determined for road elements obtained as extracted information from map data in S60. If the road width is wide (YES), the presence or absence of a sidewalk is determined in S61. If a sidewalk is present (YES), a virtual object is placed on the sidewalk in S62. If a sidewalk is not present (NO), a virtual object is placed on the road edge in S63. If the road width is narrow (NO) as determined in S60, a virtual object is placed within the road in S64. The virtual objects in S62 to S64 are placed at positions along the traveled course to measure the traveled distance.

[0045] FIG. 7 is a flowchart of the course object generation process (S25) in FIG. 7. In FIG. 7, the system determines whether the competitor is far ahead in S70, or whether the course is set to return in the opposite direction from the user's driving direction in S71, in both cases determining whether the competitor is out of the user's field of view. If the competitor is within the user's field of view (NO in both S70 and S71), a course object conforming to the road is generated in S72, and if the competitor is out of the user's field of view (YES in either S70 or S71), a pseudo course object is created in S73. The pseudo course object is a course object that reflects only the sense of distance from the competitor and is unrelated to roads in real space.

[0046] Next, specific examples of displays in this embodiment will be described with reference to FIGS. 8 to 12. FIG. 8 shows a display example in which a competitor is ahead of the user but is visible without being obscured by real objects. In FIG. 8, the left side shows a display image 50 of the HMD seen by the user, and the right side shows a corresponding map 60. Furthermore, in FIG. 8, 51 and 61 are the user, 52 and 62 are competitors (virtual objects, with 52 being an avatar in particular), and 63 is the running course. On the left side of FIG. 8, user 51 is not included in display image 50, but is shown for reference to indicate the user's position. Furthermore, an image of competitor 52 is displayed as a virtual object.

[0047] Figure 9 shows a display example in which a competitor is ahead of the user but is hidden by a building, which is a real object, and cannot be seen. In Figure 9, the same components as in Figure 8 are given the same reference numerals, and their explanation will be omitted. Figure 9 differs from Figure 8 in that a course object 54 and a distance information object 55, which are auxiliary information objects, have been added.

[0048] On the left side of Figure 9, a competitor 52, which is a virtual object, is hidden by a building, which is a real object, and would normally be invisible, but unlike the case of Figure 8, for example, the display color and drawing style are changed so that the user does not lose sight of the competitor. Also, a course object 54 is partially hidden by a building, which is a real object, but the drawing style of the hidden part is changed from that of the unhidden part to guide the user along the running course. Furthermore, a distance information object 55 is displayed to allow the user to get an accurate sense of distance.

[0049] In this way, by changing the rendering format of the hidden parts of the virtual object that is the competitor, or by displaying the running course as a virtual course as a course object and changing the rendering format of the hidden parts of the course object, it is possible to enhance the user's sense of realism, which can be useful in improving the runner's performance.

[0050] Figure 10 shows an example of placing a virtual object at a specific position. In Figure 10, the same components as in Figure 8 are assigned the same reference numerals, and their description will be omitted. In Figure 10, if the road width of the driving course is wide and a sidewalk 56 is installed, competitor 52, which is a virtual object, is placed on sidewalk 56 so that competitor 52, which is a virtual object, does not overlap with obstacles such as cars on the roadway.

[0051] Fig. 11 is a display example in which a competitor is following behind the user. In Fig. 11, the same components as in Fig. 8 are assigned the same reference numerals, and their description will be omitted. In Fig. 11, a rearview mirror object 57 is displayed at a specific position, for example, at the top, of the display image 50 of the HMD. By displaying the competitor's virtual object and distance information object within the rearview mirror object 57, it becomes easier to recognize that the competitor is following behind the user.

[0052] FIG. 12 shows an example display in which a competitor is significantly ahead of the user, or the running course is meandering as shown on the map on the right side of FIG. 12 and is outside the user's field of view (the display range of the HMD image), i.e., the virtual object is out of the viewable range. In FIG. 12, the same components as in FIG. 9 are denoted by the same reference numerals, and their description will be omitted. As shown on the left side of FIG. 12, a pseudo course object 58 including a course object 54 is displayed, and virtual objects such as a competitor 52 and a distance information object 55 are displayed on the pseudo course object 58. The pseudo course object 58 is a three-dimensional object, e.g., using perspective, that gradually moves away from the user 51. The virtual object competitor 52 is then positioned on the pseudo course object 58 at a position equivalent to the distance the competitor is ahead, depending on the distance the competitor is ahead, so that the distance when the user visually recognizes the competitor reflects the actual distance. This improves the sense of real distance on the virtual course, which can help improve runners' performance. It is also possible to display only one of the simulated course object 58 and the competitor 52 which is a virtual object.

[0053] As described above, the virtual object display method in this embodiment includes display processing, position detection processing, map information processing, virtual object processing, auxiliary information processing, and image recognition processing. The position detection processing identifies the user's position on the map, and the map information processing extracts roads and other information that have been set in advance as a driving course. The virtual object processing generates virtual objects such as competitors. The virtual objects are assigned a relative position to the user and their placement position on the driving course. The image recognition processing recognizes real objects from images in real space, and the display processing determines whether the virtual objects are within the user's visible range and displays virtual objects outside the visible range in a form different from virtual objects within the visible range. Furthermore, the auxiliary information processing displays information such as the driving course extracted by the map information processing as auxiliary information objects.

[0054] Furthermore, the device for displaying virtual objects in this embodiment comprises a display processing unit, a position detection processing unit, a map information processing unit, a virtual object processing unit, an auxiliary information processing unit, a camera unit, and an image recognition processing unit. The position detection processing unit identifies the user's position on the map, and the map information processing unit extracts roads and other information that have been set in advance as a driving course. The virtual object processing unit generates virtual objects such as competitors. The virtual objects are assigned a relative position to the user and a placement position of the virtual object on the driving course. The image recognition processing unit recognizes real objects from images of real space obtained by the camera, and the display processing unit determines whether the virtual objects are within the user's visible range and displays virtual objects outside the visible range in a form different from virtual objects within the visible range. Furthermore, the auxiliary information processing unit displays information such as the driving course extracted by the map information processing as auxiliary information objects.

[0055] As described above, according to this embodiment, in an MR space where real space and virtual space are merged, it is possible to provide a device and a display method for displaying virtual objects that can enhance the user's sense of realism by changing the rendering form so that the user does not lose sight of the virtual object even when the virtual object is not visible. Furthermore, by rendering virtual objects outside the visible range as pseudo objects, it is possible to enhance the user's sense of realism.

[0056] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the present invention is not necessarily limited to those having all of the configurations described in the embodiments. Furthermore, devices that display virtual objects may be smartphones or car navigation systems other than HMDs.

[0057] Although the functions of the present embodiment have been described as being processed by software, some or all of them may be implemented in hardware, for example, by designing an integrated circuit. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Furthermore, some or all of the functions may be implemented by a server. The server may be, for example, a local server, a cloud server, an edge server, or an online service, as long as it can execute functions in cooperation with other components via communications. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0058] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed. [Explanation of symbols]

[0059] 1, 2: HMD, 10: camera, 11: ranging sensor, 12, 12a, 12b: projection unit, 13: screen, 18, 18a, 18b: control unit, 20: image recognition processing unit, 21, 30: communication unit, 22: map information processing unit, 23: virtual object processing unit, 24: display processing unit, 25: position detection processing unit, 26: auxiliary information processing unit, 27: overall control unit, 31: CPU, 33: FROM, 35: MR processing program, 50: display image, 60: map, 51, 61: user, 52: competitor (virtual object), 62: competitor, 63: driving course, 54: course object, 55: distance information object, 56: sidewalk, 57: rearview mirror object, 58: pseudo course object

Claims

1. a map information processing step of extracting, from the map data, a first map element corresponding to the location information and a second map element corresponding to a predetermined pattern; a virtual object processing step of placing a virtual object on a real object in real space corresponding to the first map element; an auxiliary information processing step of generating a map element object corresponding to the second map element; a display processing step of rendering the virtual object and the map element object in different ways depending on whether the part of the virtual object and the part of the map element object is located in front of or behind the real object; a display step of displaying the virtual object and the map element object processed in the display processing step in a superimposed manner on the real space;

2. The method for displaying a virtual object according to claim 1 , the virtual object is a competitor in a virtual space; the map element object is a course object, The display method for virtual objects is characterized in that the display step displays the competitors and the course objects superimposed on the real space.

3. The method for displaying a virtual object according to claim 2, the map element object includes a distance information object, The display method for virtual objects is characterized in that the display step displays the competitor, the course object, and the distance information object in a superimposed manner on the real space.

4. The method for displaying a virtual object according to claim 1 , the virtual object is a competitor in a virtual space; the map element object is a distance information object, The display step comprises displaying the competitor and the distance information object in a superimposed manner at specific positions in the real space.

5. The method for displaying a virtual object according to claim 2, the auxiliary information processing step generates a pseudo course object as the map element object when the competitor is outside the display range of the image displayed in the display step; The display step is a method for displaying a virtual object, characterized in that the competitor is placed on the pseudo course object and the sense of distance to the competitor as perceived by the user is displayed so as to correspond to the distance between the user and the competitor.

6. a map information processing unit that extracts, from the map data, a first map element corresponding to the location information and a second map element corresponding to a predetermined pattern; a virtual object processing unit that places a virtual object on a real object in real space that corresponds to the first map element; an auxiliary information processing unit that generates a map element object corresponding to the second map element; a display processing unit that uses different drawing methods for the virtual object and the map element object between portions located in front of and behind the real object, and superimposes the virtual object and the map element object drawn using the different drawing methods on the real space.

7. 7. The device for displaying a virtual object according to claim 6, the virtual object is a competitor in a virtual space; the map element object is a course object, The display processing unit displays the competitors and the course objects in a superimposed manner on the real space.

8. The device for displaying a virtual object according to claim 7, the map element object includes a distance information object, The display processing unit displays the competitor, the course object, and the distance information object in a superimposed manner on the real space.

9. 7. The device for displaying a virtual object according to claim 6, the virtual object is a competitor in a virtual space; the map element object is a distance information object, The display processing unit displays the competitor and the distance information object in a superimposed manner at specific positions in the real space.

10. The device for displaying a virtual object according to claim 7, the auxiliary information processing unit generates a pseudo course object as the map element object when the competitor is outside the display range of the image displayed by the display processing unit; The display processing unit places the competitor on the pseudo course object and displays the competitor's sense of distance as perceived by the user in accordance with the distance between the user and the competitor.

Images