Image processing for generating augmented reality image

Abstract

An augmented reality system according to the present invention generates a virtual viewpoint image from a designated location in a virtual world, determines whether there is overlap in the virtual viewpoint image between a simulated object in the virtual world that corresponds to a real object in the real world and a virtual object that exists in the virtual world and is different from the simulated object, and, when it has been determined that there is overlap in the virtual viewpoint image between the simulated object and the virtual object, generates an augmented reality (AR) object for an AR image that corresponds to the virtual object in accordance with the positional relationship in the virtual viewpoint image between the simulated object and the virtual object and makes a transmissive display display the AR object.

Description

Image processing for generating augmented reality images

[0001] The present disclosure relates to image processing techniques for generating augmented reality images.

[0002] In recent years, augmented reality, a technology for superimposing a computer-generated image (virtual image) on a captured image of the real world, has been widely developed. For example, Patent Document 1 discloses a technology for applying augmented reality to a real game in which a player controls a dynamic real object such as a radio-controlled car. According to this document, the enjoyment of the real game can be enhanced by applying virtual effects to a captured image of the real game.

[0003] Japanese Patent Application Laid-Open No. 2021-126454

[0004] In the above-mentioned document, augmented reality is achieved by superimposing an image showing a visual effect on an image captured in the real world. However, it is difficult to achieve augmented reality in which an image of an object in a virtual world is simultaneously viewed while viewing the real world, rather than an image captured in the real world. For example, while it is possible to simultaneously view the real world and the virtual world by displaying an image of the virtual world on a see-through display that allows a user to see what is behind it, this may result in an unnatural scene. For example, if an image is simply displayed on a see-through display, an object in the virtual world will appear to be in front of an object in the real world, even if the object is located behind that object.

[0005] In view of these problems, the present disclosure aims to provide a technology that enables a user to properly view objects in a virtual world while visually observing the real world.

[0006] In order to solve the above problem, an augmented reality system according to one aspect of the present disclosure includes one or more processors, and at least one of the one or more processors executes a first generation process, a determination process, a second generation process, and a display process. The first generation process is a process of generating a virtual viewpoint image from a specified position in a virtual world. The determination process is a process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the virtual viewpoint image. The second generation process is a process of generating an AR object for an AR image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image. The display process is a process of displaying an AR image synthesized with the AR object on a transparent display.

[0007] In order to solve the above problem, an image processing device according to one aspect of the present disclosure includes one or more processors, and at least one of the one or more processors executes a determination process and a generation process. The determination process is a process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world. The generation process is a process of generating an AR object for an AR image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0008] In order to solve the above problem, an image processing method according to one aspect of the present disclosure includes a determination step of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world, and a generation step of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when the simulated object and the virtual object overlap in the virtual viewpoint image.

[0009] According to the technology of the present disclosure, it is possible to generate an augmented reality image in which objects in a virtual world are displayed appropriately.

[0010] FIG. 1 shows an example of the configuration of an augmented reality system according to a first embodiment. FIG. 2 shows an example of a scene viewed through AR glasses 10 by user 1. FIG. 3A shows an example of a simple display of an AR image related to a virtual vehicle C2. FIG. 3B shows another example of a simple display of an AR image related to the virtual vehicle C2. FIG. 3C shows an example of an ideal AR image in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind. FIG. 3D shows an example of a user viewpoint image in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind. FIG. 3E shows an example of an AR virtual vehicle C4 generated by an image processing device 40. FIG. 3F shows an example of an unnatural user scene. FIG. 4 shows an example of the configuration of the AR glasses 10. FIG. 5 shows an example of the configuration of a remote monitoring device 20. FIG. 6 shows an example of the configuration of a virtual space providing server 30. FIG. 7 shows an example of the configuration of an image processing device 40. FIG. 8 shows an example of a communication sequence diagram among the real vehicle C1, the AR glasses 10, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 according to the first embodiment. FIG. 9 shows an example of the configuration of an augmented reality system according to the second embodiment. FIG. 10 shows an example of the configuration of the AR glasses 11. FIG. 11 shows an example of the configuration of the image processing device 41. FIG. 12 is a diagram for explaining the procedure for generating an adjusted user viewpoint image. FIG. 13 shows an example of a communication sequence diagram among the real vehicle C1, the AR glasses 11, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 41 according to the second embodiment.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Among the components disclosed below, components having the same functions are designated by the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is one form of the present disclosure, and should be appropriately modified or changed depending on the configuration of the device and various conditions, and is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to solving the above-mentioned problems.

[0012] First Embodiment [Configuration of Augmented Reality System] FIG. 1 shows an example configuration of an augmented reality system 100 according to this embodiment. The augmented reality system 100 includes augmented reality (AR) glasses 10, a remote monitoring device 20, a virtual space providing server 30, and an image processing device 40. The AR glasses 10 are a see-through display and are worn by a user 1. The user 1 can view an image displayed by the AR glasses 10 (hereinafter, an AR image) while viewing the real world. In other words, the field of view of the user 1 includes an area of ​​the real world that is not an image displayed by the AR glasses 10 and an area of ​​the image displayed by the AR glasses 10. The virtual space providing server 30 is a server device that provides a virtual space. The virtual space providing server 30 at least constructs a virtual space and generates an image viewed from a specified position in the constructed virtual space.

[0013] The real object C1 is an object that exists in the real world. In this embodiment, as an example, the real object C1 is a vehicle in the real world (therefore, also referred to as a real vehicle C1) that runs on a circuit (see FIG. 2 described later). The real vehicle C1 moves, but its position and orientation are understood by the remote monitoring device 20 using existing technology. For example, the real vehicle C1 may acquire position information of the real vehicle C1 based on a Global Positioning System (GPS) signal or a signal from an acceleration sensor, and transmit (transmit) the information to the remote monitoring device 20. Alternatively, the remote monitoring device 20 may receive a signal from the real vehicle C1 and determine the position of the real vehicle C1. Similarly, the position and orientation of the AR glasses 10 are also understood by the remote monitoring device 20 using existing technology.

[0014] The remote monitoring device 20 functions as a device that acquires telemetry data of a predetermined monitoring target in the real world. In this embodiment, as described above, the remote monitoring device 20 monitors the circuit and acquires position information of the real vehicle C1 and the AR glasses 10. Furthermore, the remote monitoring device 20 acquires (specifies) an identifier for the real vehicle C1 and transmits the position information and identifier of the real vehicle C1 to the virtual space provision server 30. The remote monitoring device 20 also transmits position information of the AR glasses 10 to the virtual space provision server 30. Furthermore, the remote monitoring device 20 may acquire information on the direction from the position of the AR glasses 10 to the position of the real vehicle C1 and transmit this information to the virtual space provision server 30.

[0015] The virtual space providing server 30 generates an object in the virtual world corresponding to the real object C1. This object is also referred to as a simulated object C3. In this embodiment, a simulated vehicle C3, which is a simulated object corresponding to the real vehicle C1, is generated based on the position information and identifier of the real vehicle C1 obtained from the remote monitoring device 20, and its position and attitude are adjusted. This allows the simulated vehicle C3 to move in the virtual space in accordance with the movement of the real vehicle C1 in the real world (e.g., a circuit). The virtual space providing server 30 also generates an object (virtual object) in the virtual world separate from the simulated object C3. For example, a virtual vehicle C2 separate from the simulated vehicle C3 may be generated as a virtual object and moved. Note that the virtual vehicle C2 may move automatically. For example, the virtual space providing server 30 may move the virtual vehicle C2 based on the past driving records of the real vehicle C1 or other real vehicles. Alternatively, the virtual space providing server 30 may receive operation of the simulated vehicle C3 from the user 1 or another user, as in a game system, and the virtual space providing server 30 may move the virtual vehicle C2 in accordance with the operation.

[0016] The virtual space providing server 30 identifies the position of the AR glasses 10 in the virtual world (hereinafter referred to as the virtual position of the AR glasses 10) based on the position information of the AR glasses 10 obtained from the remote monitoring device 20. Here, because the user 1 is wearing the AR glasses 10, the virtual position of the AR glasses 10 and the position of the user 1 in the virtual world (hereinafter referred to as the virtual position of the user 1) can be considered to be the same. Then, the virtual space providing server 30 generates an image in the virtual world with the virtual position of the AR glasses 10 as the viewpoint (hereinafter referred to as a user viewpoint image) using computer graphics, as an image with the virtual position of the user 1 as the viewpoint. Note that the virtual space providing server 30 may also generate parallax images, that is, a user viewpoint image for the left eye and a user viewpoint image for the right eye. For example, the size of the AR glasses 10 may be registered in advance in the virtual space providing server 30, and the virtual position of the AR glasses 10 on the left eye side and the virtual position of the AR glasses 10 on the right eye side may be calculated to generate user viewpoint images for the left eye and the right eye, respectively. The image processing device 40 may generate parallax images based on the user viewpoint images. The user viewpoint images may also be generated using the viewpoint as well as the direction from the position of the AR glasses 10 to the position of the real vehicle C1 as the line of sight direction. The virtual space providing server 30 transmits the user viewpoint images to the image processing device 40.

[0017] The image processing device 40 generates an AR image to be displayed on the AR glasses 10 from the user's viewpoint image received from the virtual space providing server 30. For example, if a virtual vehicle C2 is shown in the user's viewpoint image, the image processing device 40 generates a virtual vehicle (hereinafter, AR virtual vehicle) C4 and transmits the AR virtual vehicle C4 to the AR glasses 10. The AR virtual vehicle C4 is an image of a vehicle corresponding to the virtual vehicle C2 (an AR object that is a virtual object for the AR image), and details will be described later. The AR glasses 10 display the AR image (e.g., the AR virtual vehicle C4) received from the image processing device 40. The AR glasses 10 are worn by the user 1, and the user 1 views the AR image through the AR glasses 10. In other words, this display process is equivalent to the AR glasses 10 displaying the received AR virtual vehicle C4 in the field of view of the user 1 wearing the AR glasses 10. Therefore, the user 1 can view the AR virtual vehicle C4 while visually observing a scene in the real world. It should be noted that in this disclosure, the term image is understood to include still and / or moving images.

[0018] FIG. 2 shows an example of a scene (user scene) that the user 1 sees through the AR glasses 10. The user scene 200 is a user scene of the user 1 wearing the AR glasses 10 on a circuit in the real world. The user 1 is watching a real car race on the circuit, and can see a real vehicle C1 and an AR virtual vehicle C4 corresponding to a virtual vehicle C2 through the AR glasses 10. The AR virtual vehicle C4 does not exist in the real car race, but is generated by the virtual space providing server 30 and the image processing device 40 and displayed on the AR glasses 10. This allows the user 1 to enjoy a car race involving the real vehicle C1 and the AR virtual vehicle C4 through the AR glasses 10.

[0019] Next, the AR virtual vehicle C4 will be described with reference to Figures 3A to 3E. As shown in Figure 2, when the user 1 views the real vehicle C1 and the AR virtual vehicle C4 through the AR glasses 10, the positional relationship (e.g., front-to-back relationship) between the real vehicle C1 and the AR virtual vehicle C4 viewed by the user 1 changes depending on the movement of the real vehicle C1 and the movement of the virtual vehicle C2. For example, if the real vehicle C1 and the virtual vehicle C2 are close to each other and are depicted overlapping in the user viewpoint image, a problem may arise depending on whether the real vehicle C1 or the virtual vehicle C2 is positioned in front.

[0020] To explain the AR virtual vehicle C4, FIGS. 3A and 3B show examples in which a virtual vehicle C2 is simply displayed instead of the AR virtual vehicle C4. A user view 300A in FIG. 3A is an example of a user view in which the virtual vehicle C2 is located in front and the real vehicle C1 is located behind. In the user view 300A, the virtual vehicle C2 is displayed superimposed on the real world including the real vehicle C1, which does not appear unnatural. This is because the positional relationship between the real vehicle C1 and the virtual vehicle C2 (front: virtual vehicle C2, rear: real vehicle C1) matches the superimposition relationship (front: virtual vehicle C2, rear: real vehicle C1). A user view 300B in FIG. 3B is an example of an AR image in which the real vehicle C1 is located in front and the virtual vehicle C2 is located behind. In the user view 300B, a virtual vehicle C2 located behind the real vehicle C1 is displayed superimposed on the real world including the real vehicle C1. In the user scene 300B, the virtual vehicle C2 is not visually recognized as being behind the real vehicle C1, creating an unnatural scene. This is because a see-through display such as the AR glasses 10 can only display an image as if it were in front of the scene, and the positional relationship between the real vehicle C1 and the virtual vehicle C2 (front: real vehicle C1, rear: virtual vehicle C2) does not match the superimposition relationship (front: virtual vehicle C2, rear: real vehicle C1).

[0021] 3C shows an example of an ideal user view in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind the real vehicle C1. The user view 300C in FIG. 3C is an example in which the virtual vehicle C2 is depicted so that it can be seen that it is located behind the real vehicle C1. In order to generate an AR image such as the user view 300C, in this embodiment, the image processing device 40 generates an AR virtual vehicle C4 from the user viewpoint image generated by the virtual space providing server 30.

[0022] 3D shows an example of a user-perspective image generated by the virtual space providing server 30 when the real vehicle C1 is located in front and the virtual vehicle C2 is located behind. As described above, the user-perspective image is an image in which the viewpoint is the virtual position of the AR glasses 10 in the virtual world. Because the user-perspective image 300D in FIG. 3D is generated using computer graphics, the positional relationship of the vehicles is depicted correctly. That is, the user-perspective image 300D is depicted so that the simulated vehicle C3 corresponding to the real vehicle C1 is in front and the virtual vehicle C2 is in the rear.

[0023] 3E shows an example of an AR virtual vehicle C4 generated by the image processing device 40. The image 300E in FIG. 3E is an image in which the simulated vehicle C3 has been removed (in other words, made transparent) from the user viewpoint image 300D shown in FIG. 3D, and this becomes the AR virtual vehicle C4. The image 300E (i.e., the AR virtual vehicle C4) generated by the image processing device 40 is transmitted to the AR glasses 10 and displayed by the AR glasses 10. This allows the user 1 wearing the AR glasses 10 to view the real vehicle C1 and the AR virtual vehicle C4 without feeling uncomfortable.

[0024] Below, an example of the configuration of each device (system) that constitutes the augmented reality system 100 that realizes such processing, as well as an example of a specific processing procedure, will be described.

[0025] [Configuration of AR Glasses 10] Figure 4 shows an example configuration of the AR glasses 10 according to this embodiment. The AR glasses 10 include, as hardware components, a CPU 101, a ROM 102, a RAM 103, a HDD 104, an input unit 105, a display unit 106, and a communication unit 107. The AR glasses 10 may also include an external memory. The AR glasses 10 also include, as functional components executed by the CPU 101, a display control unit 111 and an AR glasses position acquisition unit 112. Note that, in this embodiment, the AR glasses 10 are assumed to be glasses-type devices that can be worn by the user 1, but they may also be goggle-type or hat-type devices, or may be devices that are not worn by the user, such as a teleprompter.

[0026] The CPU (Central Processing Unit) 101 is configured with one or more processors and comprehensively controls the operations of the AR glasses 10. The CPU 101 may be replaced with one or more processors such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. The functional configuration of the CPU 101 will be described later.

[0027] The ROM (Read Only Memory) 102 is a nonvolatile memory that stores control programs and the like necessary for the CPU 101 to execute processing. Note that these programs may be stored in a nonvolatile memory such as a hard disk drive (HDD) 104 or a solid state drive (SSD), or in an external memory such as a removable storage medium (not shown). The RAM (Random Access Memory) 103 is a volatile memory that functions as the main memory, work area, and the like of the CPU 101. That is, when executing processing, the CPU 101 loads the necessary programs and the like from the ROM 102 into the RAM 103 and executes the programs and the like to realize various functional operations.

[0028] The HDD 104 stores, for example, various data and information required when the CPU 101 performs processing using a program. The HDD 104 also stores, for example, various data and information obtained when the CPU 101 performs processing using a program. Note that this storage may be performed using a non-volatile memory such as an SSD or an external memory such as a removable storage medium together with the HDD 104 or instead of the HDD 104.

[0029] The input unit 105 is configured to be able to accept operations by the user 1. The input unit 105 can accept, for example, operations from another communication device (e.g., a smartphone) configured to be able to communicate with the AR glasses 10, operations using gestures, and operations using voice. The display unit 106 is a transmissive display that allows the user to see what is behind it. The type of transmissive display is not particularly limited, and may be a transmissive organic EL display, a transmissive inorganic EL display, a transmissive LCD (liquid crystal display), or the like.

[0030] The communication unit 107 is an interface that controls communication between the AR glasses 10 and an external device. In this embodiment, the communication unit 107 is configured to communicate with the remote monitoring device 20 and the image processing device 40, for example, using the Internet and a wireless LAN (a wireless local area network compliant with the IEEE 802.11 series).

[0031] Next, a functional configuration of the CPU 101 of the AR glasses 10 will be described. The display control unit 111 controls display on the display unit 106. In this embodiment, the display control unit 111 displays the AR virtual vehicle C4 received from the image processing device 40 via the communication unit 107 on the display unit 106. When the display control unit 111 receives parallax images (images for the left eye and right eye with different parallax), the display control unit 111 may control the display unit 106 to display each parallax image in an area of ​​the display unit 106 for displaying the parallax image.

[0032] The AR glasses position acquisition unit 112 acquires information (position information) indicating the position of the AR glasses 10 from a GPS signal or the like received via the communication unit 107. The AR glasses position acquisition unit 112 transmits the acquired position information of the AR glasses 10 to the remote monitoring device 20 via the communication unit 107.

[0033] 5 shows an example of the configuration of the remote monitoring device 20 according to this embodiment. The remote monitoring device 20 includes, as its hardware configuration, a CPU 201, a ROM 202, a RAM 203, a HDD 204, and a communication unit 205. The remote monitoring device 20 also includes, as its functional configuration executed by the CPU 201, a real vehicle management unit 211 and an AR glass position management unit 212.

[0034] The basic configuration of the CPU 201, ROM 202, RAM 203, and HDD 204 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in Fig. 4, and therefore description thereof will be omitted. However, the RAM 203 stores vehicle identifier information 213. The vehicle identifier information 213 will be described later.

[0035] The communication unit 205 is an interface that controls communication between the remote monitoring device 20 and an external device. In this embodiment, the communication unit 205 is configured to communicate with the real vehicle C1, the AR glasses 10, and the virtual space providing server 30, for example, using the Internet and a wireless LAN.

[0036] Next, the functional configuration of the CPU 201 of the remote monitoring device 20 will be described. The real vehicle management unit 211 monitors the circuit and acquires information about the real vehicle C1 and the AR glasses 10 via the communication unit 205. For example, the real vehicle management unit 211 acquires position information of the real vehicle C1 from the real vehicle C1. As described above, the real vehicle C1 has the function of transmitting (transmitting) position information of the real vehicle C1 to the remote monitoring device 20, and the real vehicle management unit 211 can acquire this position information. Furthermore, the real vehicle management unit 211 identifies (acquires) an identifier of the real vehicle C1 that can be used in both the real world and the virtual world by referring to the vehicle identifier information 213. For example, when the real vehicle management unit 211 monitors the circuit and extracts characteristics of the real vehicle C1 (e.g., shape, size, color), the real vehicle management unit 211 can acquire an identifier corresponding to the characteristic from the vehicle identifier information 213 using known image recognition processing and identify the identifier as the identifier of the real vehicle C1. Alternatively, if the real vehicle C1 has an RFID (Radio Frequency Identification) tag attached, tag information readable from the RFID tag can be used. For example, the real vehicle management unit 211 may monitor the circuit and obtain an identifier corresponding to the tag information read from the RFID tag from the vehicle identifier information 213, and identify the identifier as the identifier of the real vehicle C1. The real vehicle management unit 211 transmits the obtained location information and identifier of the real vehicle C1 to the virtual space provision server 30 via the communication unit 205.

[0037] The AR glasses position management unit 212 acquires the position information of the AR glasses 10 from the AR glasses 10 via the communication unit 205 and transmits it to the virtual space provision server 30. The remote monitoring device 20 continues to transmit the position information and identifier of the real vehicle C1 and the position information of the AR glasses 10 to the virtual space provision server 30.

[0038] [Configuration of Virtual Space Providing Server 30] Figure 6 shows an example configuration of the virtual space providing server 30 according to this embodiment. The virtual space providing server 30 includes, as its hardware configuration, a CPU 301, a ROM 302, a RAM 303, a HDD 304, and a communication unit 305. The virtual space providing server 30 also includes, as functional components executed by the CPU 301, a virtual world management unit 311 and an image generation unit 312. The basic configuration of the CPU 301, ROM 302, RAM 303, and HDD 304 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in Figure 4, and therefore a description thereof will be omitted. However, the RAM 303 stores content information 313. The content information 313 will be described later.

[0039] The communication unit 305 is an interface that controls communication between the virtual space providing server 30 and external devices. In this embodiment, the communication unit 405 is configured to communicate with the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 via a communication network such as the Internet or a wireless LAN.

[0040] Next, the functional configuration of the CPU 301 of the virtual space providing server 30 will be described. The virtual world management unit 311 constructs a virtual space and manages the entire virtual world. For example, the virtual world management unit 311 generates a simulated vehicle C3, which is a vehicle in the virtual world corresponding to the real vehicle C1, based on the identifier of the real vehicle C1 received from the remote monitoring device 20 via the communication unit 305. In this embodiment, the content information 313 stored in the RAM 303 includes an image of the vehicle corresponding to the vehicle identifier. The virtual world management unit 311 acquires an image of the simulated vehicle C3 corresponding to the identifier of the real vehicle C1 from the content information 313. The image is an image that imitates the shape and design of the real vehicle C1, which has been generated and stored in advance. The virtual world management unit 311 then moves the simulated vehicle C3 using the position information of the real vehicle C1 received from the remote monitoring device 20 via the communication unit 305. As described above, the virtual world management unit 311 also moves the virtual vehicle C2.

[0041] The virtual world management unit 311 manages objects in the virtual world by identifiers. Specifically, the virtual world management unit 311 manages simulated vehicle C3 by the identifier of real vehicle C1 received from the remote monitoring device 20. The virtual world management unit 311 also manages virtual vehicle C2 by a newly generated identifier. Furthermore, the virtual world management unit 311 manages whether an object in the virtual world is a simulated object or a virtual object. For example, the virtual world management unit 311 manages whether a vehicle in the virtual world is a simulated vehicle or a virtual vehicle by associating it with an identifier. As an example, if the virtual vehicle is set to "0" and the simulated vehicle is set to "1," and the identifier of virtual vehicle C2 is "C2," and the identifier of simulated vehicle C3 is "C3," the virtual world management unit 311 manages the identifier of virtual vehicle C2 as "C2-0" and the identifier of simulated vehicle C3 as "C3-1."

[0042] The image generation unit 312 generates an image from a specific virtual position. For example, the image generation unit 312 identifies the virtual position of the AR glasses 10 (the position of the AR glasses 10 in the virtual world) based on position information of the AR glasses 10 received from the remote monitoring device 20 via the communication unit 305. The image generation unit 312 then generates (creates) a user's viewpoint image (an image in which the viewpoint is the virtual position of the AR glasses 10 in the virtual world) using computer graphics. The image generation unit 312 may further generate the user's viewpoint image using the direction from the position of the AR glasses 10 to the position of the real vehicle C1 as the line of sight. An example of the generated user's viewpoint image is the user's viewpoint image 300D in FIG. 3D , which is generated so that the positional relationship between the simulated vehicle C3 and the virtual vehicle C2 can be visually recognized. The image generation unit 312 transmits the generated user's viewpoint image to the image processing device 40 via the communication unit 305. Naturally, the image generation unit 312 may also generate images other than the user's viewpoint image. For example, images may be generated from various virtual positions to show the state of the virtual space. Furthermore, when an operation on the virtual vehicle C2 is accepted, an image may be generated to show the virtual space to the user who is performing the operation, and displayed on a monitor (not shown) for that purpose.

[0043] The image generation unit 312 also generates meta information including information on the area of ​​one or more vehicles included in the user-perspective image (such as coordinate information of the vehicle in the user-perspective image), its identifier, and its positional relationship (front-to-back relationship). For example, if identifiers for virtual vehicle C2 and simulated vehicle C3 are set as described above, the meta information includes the identifier "C2-0" of virtual vehicle C2 and the area of ​​virtual vehicle C2 in the user-perspective image, the identifier "C3-1" of simulated vehicle C3 and the area of ​​simulated vehicle C3 in the user-perspective image, and information on the positional relationship between virtual vehicle C2 and simulated vehicle C3. The image generation unit 312 transmits the generated meta information to the image processing device 40 via the communication unit 305. Note that, in this embodiment, the function of the image generation unit 312 is incorporated into the virtual space provision server 30; however, the function may be configured to be provided by a device separate from the virtual space provision server 30.

[0044] [Configuration of Image Processing Device 40] Fig. 7 shows an example of the configuration of the image processing device 40 according to this embodiment. The image processing device 40 includes, as a hardware configuration, a CPU 401, a ROM 402, a RAM 403, a HDD 404, and a communication unit 405. The image processing device 40 also includes an image processing unit 411 as a functional configuration executed by the CPU 401. The basic configuration of the CPU 401, ROM 402, RAM 403, and HDD 404 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in Fig. 4, and therefore description thereof will be omitted.

[0045] The communication unit 405 is an interface that controls communication between the image processing device 40 and an external device. In this embodiment, the communication unit 405 is configured to communicate with the virtual space providing server 30 and the AR glasses 10 via a communication network such as the Internet or a wireless LAN.

[0046] Next, the functional configuration of the CPU 301 of the image processing device 40 will be described. The image processing unit 411 acquires a user-perspective image and meta information from the virtual space providing server 30 via the communication unit 405. The image processing unit 411 then generates an AR virtual vehicle C4 from the user-perspective image based on the user-perspective image and the meta information. As described above, the meta information includes information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image, and the image processing unit 411 determines whether the two areas overlap, and generates the AR virtual vehicle C4 as follows based on the result of this determination.

[0047] If the area of ​​the virtual vehicle C2 and the area of ​​the simulated vehicle C3 do not overlap in the user-viewpoint image, the image processing unit 411 generates the full-size virtual vehicle C2 as the AR virtual vehicle C4. That is, based on the information on the areas of the virtual vehicle C2 and the simulated vehicle C3 included in the meta information, the image processing unit 411 deletes (in other words, makes transparent) the area other than the virtual vehicle C2 from the user-viewpoint image and generates the AR virtual vehicle C4 (the virtual vehicle C2 and the AR virtual vehicle C4 are the same).

[0048] On the other hand, if the area of ​​the virtual vehicle C2 and the area of ​​the simulated vehicle C3 overlap in the user-perspective image, the image processing unit 411 generates an AR virtual vehicle C4 taking into account the positional relationship between the virtual vehicle C2 and the simulated vehicle C3 contained in the meta information. For example, if the positional relationship between the simulated vehicle C3 and the virtual vehicle C2 contained in the meta information is such that the virtual vehicle C2 is in front and the simulated vehicle C3 is in the rear, the image processing unit 411 generates a full-size virtual vehicle C2 as the AR virtual vehicle C4. That is, based on the information on the areas of the virtual vehicle C2 and the simulated vehicle C3 contained in the meta information, the image processing unit 411 generates an AR virtual vehicle C4 by deleting the area other than the virtual vehicle C2 from the user-perspective image (the virtual vehicle C2 and the AR virtual vehicle C4 will be the same). On the other hand, if the positional relationship between the simulated vehicle C3 and the virtual vehicle C2 included in the meta information indicates that the virtual vehicle C2 is behind and the simulated vehicle C3 is in front, the image processing unit 411 generates an AR virtual vehicle C4 by deleting the area (portion) of the virtual vehicle C2 where the virtual vehicle C2 and the simulated vehicle C3 overlap from the virtual vehicle C2. That is, based on the information about the areas of the virtual vehicle C2 and the simulated vehicle C3 included in the meta information, the image processing unit 411 generates an AR virtual vehicle C4 by deleting the area of ​​the virtual vehicle C2 where the virtual vehicle C2 and the simulated vehicle C3 overlap from the user viewpoint image. Examples of the AR virtual vehicle C4 are as described with reference to Figures 3D and 3E, and the image is an image in which the area where the virtual vehicle C2 and the simulated vehicle C3 overlap is deleted from the virtual vehicle C2. The image processing unit 411 transmits the generated AR virtual vehicle C4 to the AR glasses 10 via the communication unit 405.

[0049] The AR glasses 10, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 may each have dedicated hardware for executing their respective functions, or may have some of their functions executed by hardware and the other parts executed by a computer running a program. Alternatively, all of their functions may be executed by a computer and a program.

[0050] [Processing Flow] The processing flow according to this embodiment will be described with reference to Fig. 8. Fig. 8 shows an example of a communication sequence diagram between the real vehicle C1, AR glasses 10, remote monitoring device 20, virtual space providing server 30, and image processing device 40 according to this embodiment. Note that the order of each process is not limited to the order shown in Fig. 8.

[0051] In this example, a real vehicle C1 running on a real circuit is reproduced in a virtual space as a simulated vehicle C3. Then, in the virtual space, the simulated vehicle C3 and a virtual vehicle C2, which does not correspond to the real vehicle running on the real circuit, are run. Furthermore, the user 1 simultaneously visualizes the real vehicle C1 running on the real circuit and the AR virtual vehicle C4 (corresponding to the virtual vehicle C2) through the AR glasses 10.

[0052] First, the virtual space providing server 30 generates a virtual space (S801). This generation also includes virtual objects such as a virtual vehicle C2. The remote monitoring device 20 monitors the circuit (S802). Here, the remote monitoring device 20 may extract characteristics (e.g., shape, size, color) of the real vehicle C1. Alternatively or additionally, the remote monitoring device 20 may acquire tag information from an RFID tag attached to the real vehicle C1. Furthermore, during the monitoring of the circuit (S802), the remote monitoring device 20 acquires position information of the real vehicle C1 transmitted by the real vehicle C1 and position information of the AR glasses 10 transmitted by the AR glasses 10.

[0053] Based on the information obtained by monitoring the circuit (S802), the remote monitoring device 20 acquires the identifier of the real vehicle C1 (S803). For example, the remote monitoring device 20 can acquire the identifier of the real vehicle C1 based on the characteristics and tag information of the real vehicle C1 obtained by monitoring the circuit in S901. The remote monitoring device 20 transmits the position information and identifier of the real vehicle C1 and the position information of the AR glasses 10 to the virtual space providing server 30 (S804). The processes from S802 to S804 are performed continuously.

[0054] Based on the identifier of the real vehicle C1 received from the remote monitoring device 20, the virtual space provision server 30 generates a simulated vehicle C3 in the virtual space, which is a vehicle in the virtual world corresponding to the real vehicle C1. The virtual space provision server 30 then uses the position information of the real vehicle C1 received from the remote monitoring device 20 to move the simulated vehicle C3 in the virtual space (S805). The virtual space provision server 30 also generates a user-perspective image (an image in the virtual world from the viewpoint of the virtual position of the AR glasses 10) using computer graphics based on the position information of the AR glasses 10 received from the remote monitoring device 20 (S806). The virtual space provision server 30 then generates meta information including information on the areas (positions) and identifiers of one or more vehicles included in the user-perspective image (S806). In this example, the meta information includes the identifier of the virtual vehicle C2, the identifier of the simulated vehicle C3, and information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image and the positional relationship (front-to-back relationship) between the two vehicles. The virtual space providing server 30 transmits the generated user's viewpoint image and meta information to the image processing device 40 (S807).

[0055] The image processing device 40 generates an AR virtual vehicle C4 based on the user-perspective image and meta information received from the virtual space providing server 30 (S808). As described above, the meta information includes information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image and the positional relationship between the two vehicles. The image processing device 40 determines whether the two areas overlap, and if they do, generates the AR virtual vehicle C4 taking into account the positional relationship between the two vehicles. For example, as in the user-perspective image 300D of FIG. 3D , if the areas of the virtual vehicle C2 and the simulated vehicle C3 overlap and the positions of the two vehicles are such that the simulated vehicle C3 is in front and the virtual vehicle C2 is behind, the image processing device 40 generates an AR virtual vehicle C4 as in the image 300E of FIG. 3E . That is, the image processing device 40 generates an AR virtual vehicle C4 by deleting the area of ​​the virtual vehicle C2 that overlaps with the simulated vehicle C3 from the virtual vehicle C2.

[0056] In this embodiment, it is assumed that a real object in the real world matches a simulated object in the user-perspective image generated by the virtual space providing server 30. For example, it is assumed that the position (and shape) of the simulated vehicle C3 in the user-perspective image 300D matches the position (and shape) of the real vehicle C1 as seen by the user. Therefore, the AR virtual vehicle C4 viewed through the AR glasses 10 can be extracted by adjusting the display portion of the AR virtual vehicle C4 according to the front-to-back relationship between the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image 300D. Note that in this disclosure, the term "match" is synonymous with "almost match" (e.g., the degree of match is within a predetermined range). The image processing device 40 transmits the generated AR virtual vehicle C4 and information about the area of ​​the virtual vehicle C2 included in the meta information (e.g., coordinate information about the virtual vehicle C2 in the user-perspective image) to the AR glasses 10 (S809).

[0057] The AR glasses 10 receive information about the area of ​​the virtual vehicle C2 included in the meta information (such as coordinate information about the virtual vehicle C2 in the user's viewpoint image) along with the AR virtual vehicle C4 from the image processing device 40. The AR glasses 10 then display the AR virtual vehicle C4 in the area indicated by the received information on the display unit 106 (S810). This allows the user 1 to view the real vehicle C1 and the virtual vehicle C2 in a natural positional relationship. For example, as shown in FIG. 3C , even when the virtual vehicle C2 (corresponding to the AR virtual vehicle C4) is located behind the real vehicle C1, the position of the virtual vehicle C2 relative to the real vehicle C1 does not appear unnatural.

[0058] In this way, the image processing device 40 according to this embodiment generates an AR virtual vehicle C4 to be displayed on the AR glasses 10 from the virtual vehicle C2 in an image (user viewpoint image) from the viewpoint of the user 1 in the virtual world, based on the areas and positional relationship between the virtual vehicle C2 and the simulated vehicle C3. The AR glasses 10 display the AR virtual vehicle C4 in an appropriate area of ​​the display unit. Because the AR virtual vehicle C4 only has a portion visible to the user 1, the front-to-back relationship between the virtual vehicle C2 and the simulated vehicle C3 in the user viewpoint image matches the front-to-back relationship between the virtual vehicle C2 and the real vehicle C1 viewed through the AR glasses 10. This reduces the sense of discomfort felt by the user 1. In other words, the user 1 in the real world can view an AR image in a more realistic display manner.

[0059] In this embodiment, a case where multiple vehicles move in the real world and the virtual world has been described, but as mentioned above, objects that move in the real world and the virtual world are not limited to vehicles. This embodiment can be applied to any dynamic object or static object whose front-to-back relationship can change in the real world and the virtual world.

[0060] Second Embodiment In the first embodiment, the virtual space providing server 30 generated a user viewpoint image, which is an image with the virtual position of the user 1 (the position of the user 1 in the virtual world) as the viewpoint, and an image with the virtual position of the AR glasses 10 (the position of the AR glasses 10 in the virtual world) as the viewpoint. However, depending on the orientation and posture of the user 1, the two viewpoints do not necessarily coincide. That is, the user 1 wearing the AR glasses 10 can change their height, orientation, and posture, and the viewpoint of the virtual position of the user 1 may change accordingly. That is, the discrepancy between the user viewpoint image generated by the virtual space providing server 30 and the real world as viewed by the user may be noticeable. If this discrepancy is large, an AR image in which a portion of a virtual object is removed may appear unnatural. FIG. 3F shows an example of an unnatural user viewpoint. The user viewpoint 300F in FIG. 3F is perceived as missing a portion of the AR virtual vehicle C4, which creates an uncomfortable feeling for the user. Therefore, in this embodiment, the AR glasses 10 worn by the user 1 perform an imaging process, correct a user's viewpoint image based on an image of the real world (real image) obtained by the imaging process, and generate an AR virtual vehicle from the user's viewpoint image. Below, differences from the first embodiment will be described, and common parts will not be described.

[0061] 9 shows an example of the configuration of an augmented reality system 1000 according to this embodiment. The augmented reality system 1000 according to this embodiment differs from the augmented reality system 100 in FIG. 1 in the configurations of the AR glasses 11 and the image processing device 41. The AR glasses 11 can transmit a real image P1 to the image processing device 41.

[0062] FIG. 10 shows an example configuration of the AR glasses 11 according to this embodiment. The AR glasses 11 differ from the AR glasses 10 shown in FIG. 4 in that the AR glasses 11 further include an imaging unit 108 and an imaging control unit 112. The imaging unit 108 recognizes the real world viewed by the user 1, performs imaging processing, and generates an image of the real world. The imaging unit 108 is disposed in a position where it can reproduce the world viewed by the user 1 wearing the AR glasses 10. Note that the AR glasses 10 may include multiple imaging units 108. The imaging control unit 112 controls the imaging processing by the imaging unit 108. The imaging control unit 112 can control the imaging unit 108 in accordance with an operation by the user 1 or a predetermined setting. The imaging control unit 112 can also transmit the user's field of view (e.g., real image P1) generated by the imaging unit 108 to the image processing device 40 via the communication unit 107.

[0063] FIG. 11 shows an example of the configuration of an image processing device 41 according to this embodiment. The image processing device 41 has a configuration different from the image processing device 40 shown in FIG. 7 in that it has an image processing unit 412 as a functional configuration executed by the CPU 401. The RAM 403 also stores an image adjustment model 413. The image adjustment model 413 may also be stored in the ROM 402. The image adjustment model 413 is a learning model for deep learning that is configured to input a user-perspective image generated by the virtual space providing server 30 and a real image P1 obtained by the AR glasses 10, and predict and output an adjusted user-perspective image obtained by adjusting (correcting) the orientation and / or size of the user-perspective image. The image adjustment model 413 has been trained in advance and is stored in the RAM 403 (or the ROM 402).

[0064] In this embodiment, the discrepancy between the user-perspective image generated by the virtual space providing server 30 and the real image P1 obtained by the AR glasses 10 is considered to be perceptible but not significant. In other words, the same object is displayed in the user-perspective image and the real image P1, but the position, orientation, size, etc. of the object may differ slightly. Therefore, the image adjustment model 413 corrects the position, orientation, and / or size of the object in the input user-perspective image so that the object in the user-perspective image matches the object in the real image P1. This correction is performed using an image adjustment model 413 created by deep learning. This model recognizes the target object displayed in the two input images and corrects one of the images so that the target object in one image matches the object in the other image. This model may be a known model.

[0065] The image processing unit 412 inputs the user's viewpoint image received from the virtual space providing server 30 and the real image P1 received from the AR glasses 10 into the image adjustment model 413 to obtain an adjusted user's viewpoint image. Furthermore, the image processing unit 412 according to this embodiment generates an AR virtual vehicle from the adjusted user's viewpoint image using a procedure similar to that described in the first embodiment.

[0066] An overview of the procedure for generating an adjusted user-viewpoint image according to this embodiment will be described with reference to FIG. 12 . FIG. 12 is a diagram for explaining the procedure for generating an adjusted user-viewpoint image according to this embodiment. The user-viewpoint image 1201 is a user-viewpoint image generated by the image generation unit 312 of the virtual space providing server 30, and includes a virtual vehicle C2 and a simulated vehicle C3. The real image 1202 is a real image generated by the imaging unit 108 of the AR glasses 10 worn by the user 1, and includes a real vehicle C1. The image adjustment model 413 inputs the user-viewpoint image 1201 and the real image 1202 and determines that the real vehicle C1 and the simulated vehicle C3 correspond to each other based on color and shape matches and / or identifier information, etc. The image adjustment model 413 then corrects the position, orientation, and / or size of the user-viewpoint image 1201 using the real vehicle C1 as a reference so that the simulated vehicle C3 matches the real vehicle C1, and outputs an adjusted user-viewpoint image 1203. This reduces the deviation between the adjusted user viewpoint image 1203 and the real world viewed by the user.

[0067] [Processing Flow] The processing flow according to this embodiment will be described with reference to Fig. 13. Fig. 13 shows an example of a communication sequence diagram between the real vehicle C1, AR glasses 10, remote monitoring device 20, virtual space providing server 30, and image processing device 41 according to this embodiment. Note that the order of each process is not limited to the order shown in Fig. 13. Processes similar to those in Fig. 8 are assigned the same reference numerals and descriptions thereof will be omitted.

[0068] The image processing device 41 receives the user's viewpoint image and meta information from the virtual space providing server 30 (S807), and receives the real image P1 from the AR glasses 10 (S1301). The image processing device 41 inputs the user's viewpoint image and the real image P1 into the image adjustment model 413 to generate an adjusted user's viewpoint image (S1302). Next, the image processing device 40 generates the AR virtual vehicle C4 based on the adjusted user's viewpoint image and meta information (S1303).

[0069] By this processing, even if the height, orientation, or posture of the user 1 wearing the AR glasses 10 changes and the real image and the user-viewpoint image become mismatched, the image processing device 41 generates an adjusted user-viewpoint image and generates an AR virtual vehicle based on the adjusted user-viewpoint image. Note that, in this embodiment, the image processing device 41 generates the adjusted user-viewpoint image based on the real vehicle C1, but the adjusted user-viewpoint image may also be generated based on any object such as a static object (for example, a road or a sign).

[0070] As described above, according to this embodiment, an AR virtual vehicle is generated based on an adjusted user viewpoint image generated in response to changes in the height, orientation, and posture of the user 1 wearing the AR glasses 10, and an AR image is generated using the AR virtual vehicle. This allows the AR image to be displayed on the AR glasses 10 in a manner that is more natural to the user 1, allowing the user 1 in the real world to enjoy a more realistic display mode.

[0071] Although the present embodiment describes the generation of AR images when multiple vehicles are moving in the real world and the virtual world, objects that move in the real world and the virtual world are not limited to vehicles. The present embodiment can be applied to any dynamic or static object whose front-to-back relationship can change between the real world and the virtual world.

[0072] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present disclosure. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present disclosure. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present disclosure.

[0073] (Embodiments of the Present Disclosure) The present disclosure includes the following embodiments: [1] An augmented reality system including one or more processors, wherein at least one of the one or more processors executes a first generation process of generating a virtual viewpoint image from a specified position in a virtual world, a determination process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the virtual viewpoint image, a second generation process of generating an AR object for an AR image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image, and a display process of displaying the AR object on a see-through display.

[0074] [2] The augmented reality system described in [1], wherein the second generation process includes generating the AR object by removing the area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image.

[0075] [3] The augmented reality system described in [1] or [2], wherein at least one of the one or more processors further executes a third generation process to adjust the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image, wherein the determination process includes determining whether the simulated object and the virtual object overlap in the adjusted virtual viewpoint image, and the second generation process includes generating the AR object according to the positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image.

[0076] [4] The augmented reality system described in [3], wherein in the third generation process, the deviation between the virtual object shown in the adjusted virtual viewpoint image and the object corresponding to the virtual object shown in the real image is smaller than the deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image.

[0077] [5] An augmented reality system described in any of [1] to [4], wherein the real object is a vehicle running in the real world, and the virtual object is an object simulating a vehicle running in the virtual world.

[0078] [6] An image processing device having one or more processors, wherein at least one of the one or more processors executes a determination process to determine whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world, and a generation process to generate an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0079] [7] The image processing device described in [6], wherein the generation process generates the AR object by removing the area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image.

[0080] [8] An image processing device according to [6] or [7], wherein at least one of the one or more processors further executes an adjustment process for adjusting the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image, wherein the determination process includes determining whether the simulated object and the virtual object overlap in the adjusted virtual viewpoint image, and the generation process includes generating the AR object according to the positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image.

[0081] [9] The image processing device described in [8], wherein, in the adjustment process, the deviation between the virtual object shown in the adjusted virtual viewpoint image and the object corresponding to the virtual object shown in the real image is smaller than the deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image.

[0082]

[10] An image processing method including: a determination step of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world; and a generation step of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0083]

[11] A computer-readable storage medium storing a program, the program including instructions that, when executed by one or more processors of an image processing device, cause the image processing device to execute a determination process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world, and a generation process of generating an AR (augmented reality) object for an AR image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0084] 1: User, 10: AR Glasses, 20: Remote Monitoring Device, 30: Virtual Space Providing Server, 40; 41: Image Processing Device, 101: CPU, 102: ROM, 103: RAM, 104: HDD, 105: Input Unit, 106: Display Unit, 107: Communication Unit, 108: Imaging Unit, 111: Display Control Unit, 112: AR Glasses Position Acquisition Unit, 113: Imaging Control Unit, 201: CPU, 202: ROM, 203: RAM, 204, HDD, 205 : Communication unit, 211: Real vehicle management unit, 212: AR glasses position management unit, 213: Vehicle identifier information, 301: CPU, 302: ROM, 303: RAM, 304: HDD, 305: Communication unit, 311: Virtual world management unit, 312: Image generation unit 312, 313: Content information, 401: CPU, 402: ROM, 403: RAM, 404: HDD, 405: Communication unit, 411; 412: Image processing unit, 413: Image adjustment model

Claims

1. A first generation processing unit that generates a virtual viewpoint image from a specified position in a virtual world; a determination processing unit that determines whether or not a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the virtual viewpoint image; a second generation processing unit that generates an AR object for an AR (Augmented Reality) image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image; a display processing unit that displays the AR object on a transparent display; An augmented reality system having

2. The second generation processing unit is generating the AR object by deleting an area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image; The augmented reality system of claim 1 .

3. Further comprising a third generation processing unit that adjusts the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image; the determination processing unit determines whether or not the simulated object and the virtual object overlap in the adjusted virtual viewpoint image; the second generation processing unit generates the AR object according to a positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image. The augmented reality system of claim 1 .

4. A deviation between the virtual object captured in the adjusted virtual viewpoint image and an object corresponding to the virtual object captured in the real image is smaller than a deviation between the virtual object captured in the virtual viewpoint image and the object corresponding to the virtual object captured in the real image. The augmented reality system of claim 3 .

5. the real object is a vehicle that runs in the real world; The virtual object is an object simulating a vehicle running in the virtual world. The augmented reality system of claim 1 .

6. A determination processing unit that determines whether or not a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world. a generation processing unit that generates an AR object for an AR (Augmented Reality) image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image; The image processing device includes:

7. The generation processing unit includes: generating the AR object by deleting an area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image; The image processing device according to claim 6.

8. Further comprising an adjustment processing unit that adjusts the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image, the determination processing unit determines whether or not the simulated object and the virtual object overlap in the adjusted virtual viewpoint image; the generation processing unit generates the AR object according to a positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image. The image processing device according to claim 6.

9. A deviation between the virtual object captured in the adjusted virtual viewpoint image and an object corresponding to the virtual object captured in the real image is smaller than a deviation between the virtual object captured in the virtual viewpoint image and the object corresponding to the virtual object captured in the real image. The image processing device according to claim 8.

10. An image processing method executed by an image processing device, comprising: a determining step of determining whether or not a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world; a generating step of generating an AR object for an AR (Augmented Reality) image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image; An image processing method comprising: