Image processing for generating augmented reality images

Abstract

An augmented reality system according to the present invention generates a virtual viewpoint image from a designated location in a virtual world, determines whether there is overlap in the virtual viewpoint image between a simulated object in the virtual world that corresponds to a real object in the real world and a virtual object that exists in the virtual world and is different from the simulated object, and, when it has been determined that there is overlap in the virtual viewpoint image between the simulated object and the virtual object, generates an augmented reality (AR) object for an AR image that corresponds to the virtual object in accordance with the positional relationship in the virtual viewpoint image between the simulated object and the virtual object and makes a transmissive display display the AR object.

Description

[Technical Field]

[0001] The present disclosure relates to image processing techniques for generating augmented reality images. [Background technology]

[0002] In recent years, Augmented Reality, a technology that displays images (virtual images) generated by computer graphics superimposed on images captured from the real world, has been widely developed. For example, Patent Document 1 discloses a technology that applies Augmented Reality to a real game in which dynamic real objects such as radio-controlled cars are controlled. According to this document, the enjoyment of the real game can be enhanced by applying virtual effects to images captured from the real game. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-126454 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned document, augmented reality is achieved by superimposing an image showing a visual effect on an image captured in the real world. However, it is difficult to achieve augmented reality in which an image of an object in a virtual world is simultaneously viewed while viewing the real world, rather than an image captured in the real world. For example, while it is possible to simultaneously view the real world and the virtual world by displaying an image of the virtual world on a see-through display that allows a user to see what is behind it, this may result in an unnatural scene. For example, if an image is simply displayed on a see-through display, an object in the virtual world will appear to be in front of an object in the real world, even if the object is located behind that object.

[0005] In view of these problems, the present disclosure aims to provide a technology that enables a user to properly view objects in a virtual world while visually observing the real world. [Means for solving the problem]

[0006] In order to solve the above problem, an augmented reality system according to one embodiment of the present disclosure includes one or more processors, and at least one of the one or more processors executes a first generation process, a determination process, a second generation process, and a display process. The first generation process is a process of generating a virtual viewpoint image from a specified position in a virtual world. The determination process is a process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the virtual viewpoint image. The second generation process is a process of generating an AR object for an AR image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image. The display process is a process of displaying an AR image obtained by combining the AR object on a transparent display.

[0007] In order to solve the above problem, an image processing device according to one aspect of the present disclosure includes one or more processors, At least one of the one or more processors executes a determination process and a generation process. The determination process is a process for determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world. The generation process is a process for generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0008] In order to solve the above problem, an image processing method according to one aspect of the present disclosure includes a determination step of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world, and a generation step of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when the simulated object and the virtual object overlap in the virtual viewpoint image. [Effects of the Invention]

[0009] According to the technology of the present disclosure, it is possible to generate an augmented reality image in which objects in a virtual world are displayed appropriately. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 shows an example of the configuration of an augmented reality system according to the first embodiment. [Figure 2] FIG. 2 shows an example of a scene viewed through the AR glasses 10 of the user 1. [Figure 3A]FIG. 3A shows an example in which an AR image relating to a virtual vehicle C2 is simply displayed. [Figure 3B] FIG. 3B shows another example in which an AR image relating to a virtual vehicle C2 is simply displayed. [Figure 3C] FIG. 3C shows an example of an ideal AR image in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind. [Figure 3D] FIG. 3D shows an example of a user viewpoint image in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind. [Figure 3E] FIG. 3E shows an example of an AR virtual vehicle C4 generated by the image processing device 40. [Figure 3F] FIG. 3F shows an example of an unnatural user view. [Figure 4] FIG. 4 shows an example of the configuration of the AR glasses 10. [Figure 5] FIG. 5 shows an example of the configuration of the remote monitoring device 20. [Figure 6] FIG. 6 shows an example of the configuration of the virtual space providing server 30. [Figure 7] FIG. 7 shows an example of the configuration of the image processing device 40. [Figure 8] FIG. 8 shows an example of a communication sequence diagram among the real vehicle C1, the AR glasses 10, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 according to the first embodiment. [Figure 9] FIG. 9 shows an example of the configuration of an augmented reality system according to the second embodiment. [Figure 10] FIG. 10 shows an example of the configuration of the AR glasses 11. [Figure 11] FIG. 11 shows an example of the configuration of the image processing device 41. [Figure 12] FIG. 12 is a diagram for explaining the procedure for generating an adjusted user's viewpoint image. [Figure 13] FIG. 13 shows an example of a communication sequence diagram among the real vehicle C1, the AR glasses 11, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 41 according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Among the components disclosed below, components having the same functions are designated by the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is one form of the present disclosure, and should be appropriately modified or changed depending on the configuration of the device and various conditions, and is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to solving the above-mentioned problems.

[0012] First Embodiment [Configuration of Augmented Reality System] 1 shows an example of the configuration of an augmented reality system 100 according to this embodiment. The augmented reality system 100 includes augmented reality (AR) glasses 10, a remote monitoring device 20, a virtual space providing server 30, and an image processing device 40. The AR glasses 10 are a see-through display and are worn by the user 1. The user 1 can view an image displayed by the AR glasses 10 (hereinafter referred to as an AR image) while viewing the real world. In other words, the field of view of the user 1 includes an area of ​​the real world that is not an image displayed by the AR glasses 10, and an area of ​​the image displayed by the AR glasses 10. The virtual space providing server 30 is a server device that provides a virtual space. The virtual space providing server 30 at least constructs a virtual space and generates an image of the constructed virtual space as seen from a specified position.

[0013] The real object C1 is an object that exists in the real world. In this embodiment, as an example, the real object C1 is a vehicle in the real world (therefore, also referred to as real vehicle C1) that runs on a circuit (see FIG. 2, which will be described later). The real vehicle C1 moves, but its position and orientation are understood by the remote monitoring device 20 using existing technology. For example, the real vehicle C1 may acquire position information of the real vehicle C1 based on a Global Positioning System (GPS) signal or a signal from an acceleration sensor, and transmit (transmit) this information to the remote monitoring device 20. Alternatively, the remote monitoring device 20 may receive a signal from the real vehicle C1 and determine the position of the real vehicle C1. Similarly, the position and orientation of the AR glasses 10 are also assumed to be grasped by the remote monitoring device 20 using existing technology.

[0014] The remote monitoring device 20 functions as a device that acquires telemetry data of a predetermined monitoring target in the real world. In this embodiment, as described above, the remote monitoring device 20 monitors the circuit and acquires position information of the real vehicle C1 and the AR glasses 10. Furthermore, the remote monitoring device 20 acquires (specifies) an identifier for the real vehicle C1 and transmits the position information and identifier of the real vehicle C1 to the virtual space provision server 30. The remote monitoring device 20 also transmits position information of the AR glasses 10 to the virtual space provision server 30. Furthermore, the remote monitoring device 20 may acquire information on the direction from the position of the AR glasses 10 to the position of the real vehicle C1 and transmit this information to the virtual space provision server 30.

[0015] The virtual space providing server 30 generates an object in the virtual world that corresponds to the real object C1. This object is also referred to as a simulated object C3. In this embodiment, based on the position information and identifier of the real vehicle C1 obtained from the remote monitoring device 20, the server generates a simulated vehicle C3, which is a simulated object corresponding to the real vehicle C1, and adjusts its position and attitude. This allows the simulated vehicle C3 to move in the virtual space in accordance with the movement of the real vehicle C1 in the real world (for example, on a race track). Furthermore, the virtual space providing server 30 generates an object (virtual object) in the virtual world, separate from the simulated object C3. For example, a virtual vehicle C2 separate from the simulated vehicle C3 may be generated as a virtual object and moved. The virtual vehicle C2 may also move automatically. For example, the virtual space providing server 30 may move the virtual vehicle C2 in accordance with the past driving records of the real vehicle C1 or other real vehicles. Alternatively, the virtual space providing server 30 may receive an operation of the simulated vehicle C3 from the user 1 or other users, as in a game system, and move the virtual vehicle C2 in accordance with the operation.

[0016] The virtual space providing server 30 identifies the position of the AR glasses 10 in the virtual world (hereinafter referred to as the virtual position of the AR glasses 10) based on the position information of the AR glasses 10 obtained from the remote monitoring device 20. Here, because the user 1 is wearing the AR glasses 10, the virtual position of the AR glasses 10 and the position of the user 1 in the virtual world (hereinafter referred to as the virtual position of the user 1) can be considered to be the same. Then, the virtual space providing server 30 generates an image in the virtual world with the virtual position of the AR glasses 10 as the viewpoint (hereinafter referred to as the user viewpoint image) using computer graphics, as an image with the virtual position of the user 1 as the viewpoint. The virtual space providing server 30 may generate parallax images, that is, a user viewpoint image for the left eye and a user viewpoint image for the right eye. For example, the size of the AR glasses 10 may be registered in advance in the virtual space providing server 30, and the virtual position of the left eye side of the AR glasses 10 and the virtual position of the right eye side of the AR glasses 10 may be determined to generate the user viewpoint images for the left eye and the right eye, respectively. The image processing device 40 may generate parallax images based on the user viewpoint images. The user's viewpoint image may also be generated using the direction from the position of the AR glasses 10 to the position of the real vehicle C1 as the line of sight, along with the viewpoint. The virtual space providing server 30 transmits the user's viewpoint image to the image processing device 40.

[0017] The image processing device 40 generates an AR image to be displayed on the AR glasses 10 from the user's viewpoint image received from the virtual space providing server 30. For example, if a virtual vehicle C2 appears in the user's viewpoint image, the image processing device 40 generates a virtual vehicle (hereinafter, AR virtual vehicle) C4 and transmits the AR virtual vehicle C4 to the AR glasses 10. The AR virtual vehicle C4 is an image of a vehicle (an AR object that is a virtual object for an AR image) corresponding to the virtual vehicle C2, and details will be described later. The AR glasses 10 display the AR image (for example, the AR virtual vehicle C4) received from the image processing device 40. The AR glasses 10 are worn by the user 1, and the user 1 views the AR image through the AR glasses 10. In other words, this display process is equivalent to the AR glasses 10 displaying the received AR virtual vehicle C4 in the field of view of the user 1 wearing the AR glasses 10. Therefore, the user 1 can see the AR virtual vehicle C4 while viewing the scene in the real world. It should be noted that in this disclosure, the term image is understood to include still and / or moving images.

[0018] FIG. 2 shows an example of a scene (user scene) that the user 1 sees through the AR glasses 10. The user scene 200 is a user scene of the user 1 wearing the AR glasses 10 on a circuit in the real world. The user 1 is watching a real car race on the circuit, and can see a real vehicle C1 and an AR virtual vehicle C4 corresponding to a virtual vehicle C2 through the AR glasses 10. The AR virtual vehicle C4 does not exist in the real car race, but is generated by the virtual space providing server 30 and the image processing device 40 and displayed on the AR glasses 10. This allows the user 1 to enjoy a car race involving the real vehicle C1 and the AR virtual vehicle C4 through the AR glasses 10.

[0019] Next, the AR virtual vehicle C4 will be described with reference to FIGS. 3A to 3E. 2, when the user 1 views the real vehicle C1 and the AR virtual vehicle C4 through the AR glasses 10, the positional relationship (e.g., front-to-back relationship) between the real vehicle C1 and the AR virtual vehicle C4 viewed by the user 1 changes depending on the movement of the real vehicle C1 and the movement of the virtual vehicle C2. For example, if the real vehicle C1 and the virtual vehicle C2 are close to each other and are depicted overlapping in the user viewpoint image, a problem may arise depending on whether the real vehicle C1 or the virtual vehicle C2 is located in front.

[0020] To explain the AR virtual vehicle C4, an example in which the virtual vehicle C2 is simply displayed instead of the AR virtual vehicle C4 is shown in FIGS. 3A and 3B. 3A is an example of a user view in which the virtual vehicle C2 is located in front and the real vehicle C1 is located behind. In the user view 300A, the virtual vehicle C2 is displayed superimposed on the real world including the real vehicle C1, which does not look unnatural. This is because the positional relationship between the real vehicle C1 and the virtual vehicle C2 (front: virtual vehicle C2, rear: real vehicle C1) matches the superimposition relationship (front: virtual vehicle C2, rear: real vehicle C1). The user scene 300B in FIG. 3B is an example of an AR image in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind the real vehicle C1. The user scene 300B is an example in which a virtual vehicle C2 located behind the real vehicle C1 is superimposed on the real world including the real vehicle C1. The user scene 300B is an unnatural scene in which the virtual vehicle C2 cannot be visually recognized as being behind the real vehicle C1. This is because a see-through display such as the AR glasses 10 can only display an image as if it is in front of the scene, and the positional relationship between the real vehicle C1 and the virtual vehicle C2 (front: real vehicle C1, rear: virtual vehicle C2) does not match the superimposition relationship (front: virtual vehicle C2, rear: real vehicle C1).

[0021] 3C shows an example of an ideal user view in which a real vehicle C1 is located in front and a virtual vehicle C2 is located behind the real vehicle C1. The user view 300C in FIG. 3C is an example in which the virtual vehicle C2 is depicted so that it can be seen that it is located behind the real vehicle C1. In order to generate an AR image such as the user view 300C, in this embodiment, the image processing device 40 generates an AR virtual vehicle C4 from the user viewpoint image generated by the virtual space providing server 30.

[0022] FIG. 3D shows an example of a user-perspective image generated by the virtual space providing server 30 when the real vehicle C1 is located in front and the virtual vehicle C2 is located behind. As described above, the user-perspective image is an image viewed from the virtual position of the AR glasses 10 in the virtual world. Because the user-perspective image 300D in FIG. 3D is generated using computer graphics, the positional relationship of the vehicles is depicted correctly. That is, the user-perspective image 300D depicts the simulated vehicle C3 corresponding to the real vehicle C1 in front and the virtual vehicle C2 in the rear.

[0023] FIG. 3E shows an example of an AR virtual vehicle C4 generated by the image processing device 40. The image 300E in FIG. 3E is an image in which the simulated vehicle C3 has been removed (in other words, made transparent) from the user viewpoint image 300D shown in FIG. 3D, and this becomes the AR virtual vehicle C4. The image 300E generated by the image processing device 40 (i.e., the AR virtual vehicle C4) is transmitted to the AR glasses 10 and displayed by the AR glasses 10. This allows the user 1 wearing the AR glasses 10 to see the real vehicle C1 and the AR virtual vehicle C4 without any sense of incongruity.

[0024] Below, an example of the configuration of each device (system) that constitutes the augmented reality system 100 that realizes such processing, as well as an example of a specific processing procedure, will be described.

[0025] [Configuration of AR Glasses 10] 4 shows an example configuration of the AR glasses 10 according to this embodiment. The AR glasses 10 include, as hardware components, a CPU 101, a ROM 102, a RAM 103, an HDD 104, an input unit 105, a display unit 106, and a communication unit 107. The AR glasses 10 may also include an external memory. The AR glasses 10 also include, as functional components executed by the CPU 101, a display control unit 111 and an AR glasses position acquisition unit 112. In this embodiment, the AR glasses 10 are assumed to be glasses-type devices that can be worn by the user 1, but they may also be goggle-type or hat-type devices, or may be devices that are not worn by the user, such as a prompter.

[0026] The CPU (Central Processing Unit) 101 is configured with one or more processors and comprehensively controls the operations of the AR glasses 10. The CPU 101 may be replaced with one or more processors such as an ASIC (Application specific integrated circuit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The functional configuration of the CPU 101 will be described later.

[0027] The ROM (Read Only Memory) 102 is a nonvolatile memory that stores control programs and the like required for the CPU 101 to execute processing. Note that the programs may be stored in a nonvolatile memory such as an HDD (Hard Disk Drive) 104 or an SSD (Solid State Drive), or in an external memory such as a removable storage medium (not shown). RAM (Random Access Memory) 103 is a volatile memory that functions as the main memory, work area, etc. of CPU 101. That is, when executing a process, CPU 101 loads necessary programs, etc. from ROM 102 into RAM 103 and executes the programs, etc. to realize various functional operations.

[0028] The HDD 104 stores, for example, various data and various information required when the CPU 101 performs processing using a program. The HDD 104 also stores, for example, various data and various information obtained when the CPU 101 performs processing using a program. Note that this storage may be performed using an external memory such as a nonvolatile memory such as an SSD or a removable storage medium together with the HDD 104 or instead of the HDD 104.

[0029] The input unit 105 is configured to be able to accept operations by the user 1. The input unit 105 can accept, for example, operations on another communication device (for example, a smartphone) configured to be able to communicate with the AR glasses 10, operations using gestures, and operations using voice. The display unit 106 is a transmissive display that allows the user to see through to what is behind it. The type of transmissive display is not particularly limited, and may be a transmissive organic EL display, a transmissive inorganic EL display, a transmissive LCD (liquid crystal display), or the like.

[0030] The communication unit 107 is an interface that controls communication between the AR glasses 10 and an external device. In this embodiment, the communication unit 107 is configured to communicate with the remote monitoring device 20 and the image processing device 40, for example, using the Internet and a wireless LAN (a wireless local area network conforming to the IEEE802.11 series).

[0031] Next, the functional configuration of the CPU 101 of the AR glasses 10 will be described. The display control unit 111 controls display on the display unit 106. In this embodiment, the display control unit 111 displays the AR virtual vehicle C4 received from the image processing device 40 via the communication unit 107 on the display unit 106. When the display control unit 111 receives parallax images (images for the left eye and the right eye having different parallaxes), the display control unit 111 may control the display unit 106 to display each parallax image in an area of ​​the display unit 106 for displaying the parallax image.

[0032] The AR glass position acquisition unit 112 acquires information (position information) indicating the position of the AR glasses 10 from a GPS signal or the like received via the communication unit 107. The AR glass position acquisition unit 112 transmits the acquired position information of the AR glasses 10 to the remote monitoring device 20 via the communication unit 107.

[0033] [20 remote monitoring devices] 5 shows an example of the configuration of the remote monitoring device 20 according to this embodiment. The remote monitoring device 20 includes, as a hardware configuration, a CPU 201, a ROM 202, a RAM 203, an HDD 204, and a communication unit 205. The remote monitoring device 20 also includes, as functional configurations executed by the CPU 201, a real vehicle management unit 211 and an AR glasses position management unit 212.

[0034] The basic configuration of the CPU 201, ROM 202, RAM 203, and HDD 204 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in Fig. 4, and therefore description thereof will be omitted. However, the RAM 203 stores vehicle identifier information 213. The vehicle identifier information 213 will be described later.

[0035] The communication unit 205 is an interface that controls communication between the remote monitoring device 20 and an external device. In this embodiment, the communication unit 205 is configured to communicate with the real vehicle C1, the AR glasses 10, and the virtual space providing server 30, for example, using the Internet and a wireless LAN.

[0036] Next, the functional configuration of the CPU 201 of the remote monitoring device 20 will be described. The real vehicle management unit 211 monitors the circuit and acquires information about the real vehicle C1 and the AR glasses 10 via the communication unit 205. For example, the real vehicle management unit 211 acquires position information of the real vehicle C1 from the real vehicle C1. As described above, the real vehicle C1 has the function of transmitting (transmitting) the position information of the real vehicle C1 to the remote monitoring device 20, and the real vehicle management unit 211 can acquire this position information. Furthermore, the real vehicle management unit 211 identifies (acquires) an identifier for the real vehicle C1 that can be used in both the real world and the virtual world, by referring to the vehicle identifier information 213. For example, when the real vehicle management unit 211 extracts characteristics (e.g., shape, size, color) of the real vehicle C1 by monitoring the circuit, it may acquire an identifier corresponding to the characteristics from the vehicle identifier information 213 using a known image recognition process and identify the identifier as the identifier of the real vehicle C1. Alternatively, if an RFID (Radio Frequency Identification) tag is attached to the real vehicle C1, tag information that can be read from the RFID tag may be used. For example, the real vehicle management unit 211 may acquire an identifier corresponding to tag information read from the RFID tag by monitoring the circuit from the vehicle identifier information 213 and identify the identifier as the identifier of the real vehicle C1. The real vehicle management unit 211 transmits the acquired position information and identifier of the real vehicle C1 to the virtual space provision server 30 via the communication unit 205.

[0037] The AR glasses position management unit 212 acquires the position information of the AR glasses 10 from the AR glasses 10 via the communication unit 205 and transmits it to the virtual space providing server 30. The remote monitoring device 20 continues to transmit the position information and identifier of the real vehicle C1 and the position information of the AR glasses 10 to the virtual space providing server 30.

[0038] [Configuration of virtual space providing server 30] 6 shows an example of the configuration of the virtual space providing server 30 according to this embodiment. The virtual space providing server 30 includes, as its hardware configuration, a CPU 301, a ROM 302, a RAM 303, a HDD 304, and a communication unit 305. The virtual space providing server 30 also includes, as functional components executed by the CPU 301, a virtual world management unit 311 and an image generation unit 312. The basic configuration of the CPU 301, ROM 302, RAM 303, and HDD 304 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in FIG. 4, and therefore a description thereof will be omitted. However, the RAM 303 stores content information 313. The content information 313 will be described later.

[0039] The communication unit 305 is an interface that controls communication between the virtual space providing server 30 and external devices. In this embodiment, the communication unit 405 is configured to communicate with the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 via a communication network such as the Internet or a wireless LAN.

[0040] Next, the functional configuration of the CPU 301 of the virtual space providing server 30 will be described. The virtual world management unit 311 constructs a virtual space and manages the entire virtual world. For example, the virtual world management unit 311 generates a simulated vehicle C3, which is a vehicle in the virtual world that corresponds to the real vehicle C1, based on the identifier of the real vehicle C1 received from the remote monitoring device 20 via the communication unit 305. In this embodiment, the content information 313 stored in the RAM 303 includes an image of the vehicle that corresponds to the vehicle identifier. The virtual world management unit 311 acquires an image of the simulated vehicle C3 that corresponds to the identifier of the real vehicle C1 from the content information 313. The image is an image that imitates the shape and design of the real vehicle C1, which has been generated and saved in advance. The virtual world manager 311 then uses the position information of the real vehicle C1 received from the remote monitoring device 20 via the communication unit 305 to move the simulated vehicle C3. Furthermore, the virtual world management unit 311 operates the virtual vehicle C2 as described above.

[0041] The virtual world management unit 311 manages objects in the virtual world by identifiers. Specifically, the virtual world management unit 311 manages simulated vehicle C3 by the identifier of real vehicle C1 received from the remote monitoring device 20. The virtual world management unit 311 also manages virtual vehicle C2 by a newly generated identifier. Furthermore, the virtual world management unit 311 manages whether an object in the virtual world is a simulated object or a virtual object. For example, the virtual world management unit 311 manages whether a vehicle in the virtual world is a simulated vehicle or a virtual vehicle by associating it with an identifier. As an example, if the virtual vehicle is set to "0" and the simulated vehicle is set to "1," and the identifier of virtual vehicle C2 is "C2," and the identifier of simulated vehicle C3 is "C3," the virtual world management unit 311 manages the identifier of virtual vehicle C2 as "C2-0" and the identifier of simulated vehicle C3 as "C3-1."

[0042] The image generation unit 312 generates an image from a specific virtual position. For example, the image generation unit 312 identifies the virtual position of the AR glasses 10 (the position of the AR glasses 10 in the virtual world) based on the position information of the AR glasses 10 received from the remote monitoring device 20 via the communication unit 305. Then, the image generation unit 312 generates (creates) a user's viewpoint image (an image in which the virtual position of the AR glasses 10 in the virtual world is used as the viewpoint) using computer graphics. The image generation unit 312 may further generate the user's viewpoint image using the direction from the position of the AR glasses 10 to the position of the real vehicle C1 as the line of sight. An example of the generated user's viewpoint image is the user's viewpoint image 300D in FIG. 3D , which is generated so that the positional relationship between the simulated vehicle C3 and the virtual vehicle C2 can be visually recognized. The image generation unit 312 transmits the generated user's viewpoint image to the image processing device 40 via the communication unit 305. Naturally, the image generation unit 312 may also generate images other than the user's viewpoint image. For example, images from various virtual positions may be generated to show the state of the virtual space. Furthermore, when an operation on the virtual vehicle C2 is accepted, an image may be generated to show the virtual space to the user who is performing the operation, and displayed on a monitor (not shown) for that purpose.

[0043] The image generation unit 312 also generates meta information including the area of ​​one or more vehicles included in the user-perspective image (such as coordinate information of the vehicle in the user-perspective image), its identifier, and information on its positional relationship (front-to-back relationship). For example, if identifiers for virtual vehicle C2 and simulated vehicle C3 are set as described above, the meta information includes the identifier "C2-0" of virtual vehicle C2 and the area of ​​virtual vehicle C2 in the user-perspective image, the identifier "C3-1" of simulated vehicle C3 and the area of ​​simulated vehicle C3 in the user-perspective image, and information on the positional relationship between virtual vehicle C2 and simulated vehicle C3. The image generation unit 312 transmits the generated meta information to the image processing device 40 via the communication unit 305. In this embodiment, the function of the image generation unit 312 is incorporated into the virtual space provision server 30, but the function may be configured to be provided in a device separate from the virtual space provision server 30.

[0044] [Configuration of image processing device 40] 7 shows an example of the configuration of an image processing device 40 according to this embodiment. The image processing device 40 includes, as a hardware configuration, a CPU 401, a ROM 402, a RAM 403, a HDD 404, and a communication unit 405. The image processing device 40 also includes an image processing unit 411 as a functional configuration executed by the CPU 401. The basic configuration of the CPU 401, ROM 402, RAM 403, and HDD 404 is similar to that of the CPU 101, ROM 102, RAM 103, and HDD 104 in FIG. 4, and therefore a description thereof will be omitted.

[0045] The communication unit 405 is an interface that controls communication between the image processing device 40 and an external device. In this embodiment, the communication unit 405 is configured to communicate with the virtual space providing server 30 and the AR glasses 10 via a communication network such as the Internet or a wireless LAN.

[0046] Next, the functional configuration of the CPU 301 of the image processing device 40 will be described. The image processing unit 411 acquires a user-perspective image and meta information from the virtual space providing server 30 via the communication unit 405. Then, the image processing unit 411 generates an AR virtual vehicle C4 from the user-perspective image based on the user-perspective image and the meta information. As described above, the meta information includes information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image, and the image processing unit 411 determines whether the two areas overlap, and generates the AR virtual vehicle C4 as follows based on the result of the determination.

[0047] If the area of ​​the virtual vehicle C2 and the area of ​​the simulated vehicle C3 do not overlap in the user viewpoint image, the image processing unit 411 generates the full-size virtual vehicle C2 as the AR virtual vehicle C4. That is, the image processing unit 411 generates an image as the AR virtual vehicle C4 by deleting (in other words, making transparent) the area other than the virtual vehicle C2 from the user viewpoint image based on the information on the areas of the virtual vehicle C2 and the simulated vehicle C3 included in the meta information (the virtual vehicle C2 and the AR virtual vehicle C4 will be the same).

[0048] On the other hand, if the area of ​​the virtual vehicle C2 and the area of ​​the simulated vehicle C3 overlap in the user viewpoint image, the image processing unit 411 generates the AR virtual vehicle C4 taking into account the positional relationship between the virtual vehicle C2 and the simulated vehicle C3 contained in the meta information. For example, if the positional relationship between simulated vehicle C3 and virtual vehicle C2 included in the meta information is such that virtual vehicle C2 is in front and simulated vehicle C3 is in the rear, image processing unit 411 generates a full-size virtual vehicle C2 as AR virtual vehicle C4. That is, based on the information on the areas of virtual vehicle C2 and simulated vehicle C3 included in the meta information, image processing unit 411 deletes the area other than virtual vehicle C2 from the user viewpoint image and generates AR virtual vehicle C4 (virtual vehicle C2 and AR virtual vehicle C4 will be the same). On the other hand, if the positional relationship between the simulated vehicle C3 and the virtual vehicle C2 included in the meta information indicates that the virtual vehicle C2 is behind and the simulated vehicle C3 is in front, the image processing unit 411 generates an image as the AR virtual vehicle C4 by deleting the area (portion) of the virtual vehicle C2 where the virtual vehicle C2 and the simulated vehicle C3 overlap from the virtual vehicle C2. That is, based on the information on the areas of the virtual vehicle C2 and the simulated vehicle C3 included in the meta information, the image processing unit 411 generates an image as the AR virtual vehicle C4 by deleting the area of ​​the virtual vehicle C2 where the virtual vehicle C2 and the simulated vehicle C3 overlap from the user viewpoint image. Examples of the AR virtual vehicle C4 are as described with reference to FIGS. 3D and 3E, and the image is obtained by deleting the area of ​​the virtual vehicle C2 where the virtual vehicle C2 and the simulated vehicle C3 overlap from the virtual vehicle C2. The image processing unit 411 transmits the generated AR virtual vehicle C4 to the AR glasses 10 via the communication unit 405.

[0049] The AR glasses 10, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 may each have dedicated hardware for executing their respective functions, or may have some of their functions executed by hardware and the other parts executed by a computer running a program. Alternatively, all of their functions may be executed by a computer and a program.

[0050] [Processing flow] The flow of processing according to this embodiment will be described with reference to Fig. 8. Fig. 8 shows an example of a communication sequence diagram among the real vehicle C1, the AR glasses 10, the remote monitoring device 20, the virtual space providing server 30, and the image processing device 40 according to this embodiment. Note that the order of each process is not limited to the order shown in Fig. 8.

[0051] In this example, a real vehicle C1 running on a real circuit is reproduced as a simulated vehicle C3 in a virtual space. Then, in the virtual space, the simulated vehicle C3 and a virtual vehicle C2, which does not correspond to the real vehicle running on the real circuit, are run. Furthermore, the user 1 simultaneously visualizes the real vehicle C1 running on the real circuit and the AR virtual vehicle C4 (corresponding to the virtual vehicle C2) through the AR glasses 10.

[0052] First, the virtual space providing server 30 generates a virtual space (S801), which also includes virtual objects such as the virtual vehicle C2. The remote monitoring device 20 monitors the circuit (S802). Here, the remote monitoring device 20 may extract characteristics (e.g., shape, size, color) of the real vehicle C1. Alternatively or additionally, the remote monitoring device 20 may obtain tag information from an RFID tag attached to the real vehicle C1. Furthermore, during the monitoring of the circuit (S802), the remote monitoring device 20 acquires the position information of the real vehicle C1 transmitted by the real vehicle C1 and the position information of the AR glasses 10 transmitted by the AR glasses 10.

[0053] Based on the information obtained by monitoring the circuit (S802), the remote monitoring device 20 acquires the identifier of the real vehicle C1 (S803). For example, the remote monitoring device 20 can acquire the identifier of the real vehicle C1 based on the characteristics and tag information of the real vehicle C1 obtained by monitoring the circuit in S901. The remote monitoring device 20 transmits the position information and identifier of the real vehicle C1 and the position information of the AR glasses 10 to the virtual space providing server 30 (S804). The processes from S802 to S804 are performed continuously.

[0054] The virtual space providing server 30 generates in the virtual space a simulated vehicle C3, which is a vehicle in the virtual world that corresponds to the real vehicle C1, based on the identifier of the real vehicle C1 received from the remote monitoring device 20. Then, the virtual space providing server 30 moves the simulated vehicle C3 in the virtual space using the position information of the real vehicle C1 received from the remote monitoring device 20 (S805). In addition, the virtual space providing server 30 generates a user viewpoint image (an image with the viewpoint being the virtual position of the AR glasses 10 in the virtual world) using computer graphics based on the position information of the AR glasses 10 received from the remote monitoring device 20 (S806). Furthermore, the virtual space providing server 30 generates meta information including information on the area (position) and identifier of one or more vehicles included in the user viewpoint image (S806). In this example, the meta information includes the identifier of the virtual vehicle C2, the identifier of the simulated vehicle C3, and information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user viewpoint image and the positional relationship (front-to-back relationship) between the two vehicles. The virtual space providing server 30 transmits the generated user's viewpoint image and meta information to the image processing device 40 (S807).

[0055] The image processing device 40 generates the AR virtual vehicle C4 based on the user-perspective image and meta information received from the virtual space providing server 30 (S808). As described above, the meta information includes information on the areas of the virtual vehicle C2 and the simulated vehicle C3 in the user-perspective image and the positional relationship between the two vehicles. The image processing device 40 determines whether the two areas overlap, and if they do, generates the AR virtual vehicle C4 taking into account the positional relationship between the two vehicles. For example, when the area of ​​the virtual vehicle C2 and the area of ​​the simulated vehicle C3 overlap, as in user viewpoint image 300D in Fig. 3D, and the positional relationship between the two vehicles is such that the simulated vehicle C3 is in front and the virtual vehicle C2 is behind, the image processing device 40 generates an AR virtual vehicle C4, as in image 300E in Fig. 3E. That is, the image processing device 40 generates an image of the virtual vehicle C2 from which the area overlapping with the simulated vehicle C3 has been deleted, as the AR virtual vehicle C4.

[0056] In this embodiment, it is assumed that real objects in the real world match with simulated objects in the user viewpoint image generated by the virtual space providing server 30. For example, it is assumed that the position (and shape) of the simulated vehicle C3 in the user viewpoint image 300D matches the position (and shape) of the real vehicle C1 as seen by the user. Therefore, by adjusting the display portion of the AR virtual vehicle C4 according to the front-to-back relationship between the virtual vehicle C2 and the simulated vehicle C3 in the user viewpoint image 300D, it is possible to extract the AR virtual vehicle C4 viewed through the AR glasses 10. In the present disclosure, the term "match" is synonymous with "approximately match" (for example, the degree of match is within a predetermined range). The image processing device 40 transmits the generated AR virtual vehicle C4 and information on the area of ​​the virtual vehicle C2 included in the meta information (such as coordinate information on the area occupied by the virtual vehicle C2 in the user viewpoint image) to the AR glasses 10 (S809).

[0057] The AR glasses 10 receive information about the area of ​​the virtual vehicle C2 included in the meta information (such as coordinate information about the virtual vehicle C2 in the user's viewpoint image) along with the AR virtual vehicle C4 from the image processing device 40. Then, the AR glasses 10 display the AR virtual vehicle C4 in the area indicated by the information on the display unit 106 (S810). This allows the user 1 to see the real vehicle C1 and the virtual vehicle C2 in a natural positional relationship. For example, as shown in FIG. 3C, even if the virtual vehicle C2 (corresponding to the AR virtual vehicle C4) is located behind the real vehicle C1, the position of the virtual vehicle C2 relative to the real vehicle C1 does not appear unnatural.

[0058] In this way, the image processing device 40 according to this embodiment generates an AR virtual vehicle C4 to be displayed on the AR glasses 10 from the virtual vehicle C2 in an image (user viewpoint image) from the viewpoint of the user 1 in the virtual world, based on the areas and positional relationship between the virtual vehicle C2 and the simulated vehicle C3. The AR glasses 10 display the AR virtual vehicle C4 in an appropriate area of ​​the display unit. Because the AR virtual vehicle C4 only has a portion visible to the user 1, the front-to-back relationship between the virtual vehicle C2 and the simulated vehicle C3 in the user viewpoint image matches the front-to-back relationship between the virtual vehicle C2 and the real vehicle C1 viewed through the AR glasses 10. This reduces the sense of discomfort felt by the user 1. In other words, the user 1 in the real world can view an AR image in a more realistic display manner.

[0059] In this embodiment, a case where multiple vehicles move in the real world and the virtual world has been described, but as mentioned above, objects that move in the real world and the virtual world are not limited to vehicles. This embodiment can be applied to any dynamic object or static object whose front-to-back relationship can change in the real world and the virtual world.

[0060] Second Embodiment In the first embodiment, the virtual space providing server 30 generated an image with the virtual position of the user 1 (the position of the user 1 in the virtual world) as the viewpoint, and an image with the virtual position of the AR glasses 10 (the position of the AR glasses 10 in the virtual world) as the viewpoint, as a user viewpoint image. However, depending on the orientation and posture of the user 1, the two viewpoints do not necessarily coincide. That is, the user 1 wearing the AR glasses 10 can change his / her height, orientation, and posture, and the viewpoint of the virtual position of the user 1 can also change accordingly. That is, the deviation between the user viewpoint image generated by the virtual space providing server 30 and the real world as viewed by the user may be large enough to be noticeable. If this deviation is large, the AR image in which a portion of the virtual object is removed may appear unnatural. Figure 3F shows an example of an unnatural user view. The user view 300F in Figure 3F is perceived as missing a portion of the AR virtual vehicle C4, which creates a sense of discomfort for the user. Therefore, in this embodiment, the AR glasses 10 worn by the user 1 perform an imaging process, correct a user viewpoint image based on an image of the real world (real image) obtained by the imaging process, and generate an AR virtual vehicle from the user viewpoint image. Below, differences from the first embodiment will be described, and common parts will not be described.

[0061] 9 shows an example of the configuration of an augmented reality system 1000 according to this embodiment. The augmented reality system 1000 according to this embodiment differs from the augmented reality system 100 in FIG. 1 in the configurations of the AR glasses 11 and the image processing device 41. The AR glasses 11 can transmit a real image P1 to the image processing device 41.

[0062] 10 shows an example of the configuration of the AR glasses 11 according to this embodiment. The AR glasses 11 are different from the AR glasses 10 shown in FIG. 4 in that they further include an imaging unit 108 and an imaging control unit 112. The imaging unit 108 recognizes the real world viewed by the user 1, performs imaging processing, and generates an image of the real world. The imaging unit 108 is disposed in a position where it can reproduce the world viewed by the user 1 wearing the AR glasses 10. Note that the AR glasses 10 may have multiple imaging units 108. The imaging control unit 112 controls the imaging process performed by the imaging unit 108. The imaging control unit 112 can control the imaging unit 108 in accordance with an operation by the user 1 or a predetermined setting. The imaging control unit 112 can also transmit the user's field of view (for example, a real image P1) generated by the imaging unit 108 to the image processing device 40 via the communication unit 107.

[0063] FIG. 11 shows an example of the configuration of an image processing device 41 according to this embodiment. The image processing device 41 has a configuration different from that of the image processing device 40 shown in FIG. 7 in that it has an image processing unit 412 as a functional configuration executed by the CPU 401. The RAM 403 also stores an image adjustment model 413. The image adjustment model 413 may be stored in the ROM 402. The image adjustment model 413 is a learning model for deep learning that is configured to input a user-viewpoint image generated by the virtual space providing server 30 and a real image P1 obtained by the AR glasses 10, and to predict and output an adjusted user-viewpoint image obtained by adjusting (correcting) the orientation and / or size of the user-viewpoint image. The image adjustment model 413 has been trained in advance and is stored in the RAM 403 (or the ROM 402).

[0064] In this embodiment, the discrepancy between the user-perspective image generated by the virtual space providing server 30 and the real image P1 obtained by the AR glasses 10 is noticeable, but is not considered to be significant. In other words, the same object is displayed in the user-perspective image and the real image P1, but the object's position, orientation, size, etc. may be slightly different. Therefore, the image adjustment model 413 corrects the position, orientation, and / or size of the object in the input user-perspective image so that the object in the user-perspective image matches the object in the real image P1. This correction is performed using an image adjustment model 413 created by deep learning. This model recognizes the target object displayed in the two input images and corrects one image so that the target object in one image matches the object in the other image. This model may be a known model.

[0065] The image processing unit 412 inputs the user's viewpoint image received from the virtual space providing server 30 and the real image P1 received from the AR glasses 10 into the image adjustment model 413 to acquire an adjusted user's viewpoint image. Furthermore, the image processing unit 412 according to this embodiment generates an AR virtual vehicle from the adjusted user's viewpoint image using the same procedure as that described in the first embodiment.

[0066] An overview of the procedure for generating an adjusted user-viewpoint image according to this embodiment will be described with reference to FIG. 12. FIG. 12 is a diagram for explaining the procedure for generating an adjusted user-viewpoint image according to this embodiment. The user-viewpoint image 1201 is a user-viewpoint image generated by the image generation unit 312 of the virtual space provision server 30, and includes a virtual vehicle C2 and a simulated vehicle C3. The real image 1202 is a real image generated by the imaging unit 108 of the AR glasses 10 worn by the user 1, and includes a real vehicle C1. The image adjustment model 413 inputs the user-viewpoint image 1201 and the real image 1202 and determines that the real vehicle C1 and the simulated vehicle C3 correspond to each other based on color and shape matches and / or identifier information. The image adjustment model 413 then corrects the position, orientation, and / or size of the user-viewpoint image 1201 using the real vehicle C1 as a reference so that the simulated vehicle C3 matches the real vehicle C1, and outputs an adjusted user-viewpoint image 1203. This reduces the deviation between the adjusted user viewpoint image 1203 and the real world as viewed by the user.

[0067] [Processing flow] The flow of processing according to this embodiment will be described with reference to Fig. 13. Fig. 13 shows an example of a communication sequence diagram of the real vehicle C1, AR glasses 10, remote monitoring device 20, virtual space providing server 30, and image processing device 41 according to this embodiment. Note that the order of each process is not limited to the order shown in Fig. 13. Processes similar to those in Fig. 8 are assigned the same reference numerals and descriptions thereof will be omitted.

[0068] The image processing device 41 receives the user's viewpoint image and meta information from the virtual space providing server 30 (S807), and receives the real image P1 from the AR glasses 10 (S1301). The image processing device 41 inputs the user's viewpoint image and the real image P1 into the image adjustment model 413 to generate an adjusted user's viewpoint image (S1302). Next, the image processing device 40 generates the AR virtual vehicle C4 based on the adjusted user's viewpoint image and meta information (S1303).

[0069] By this processing, even if the height, orientation, or posture of the user 1 wearing the AR glasses 10 changes and the real image and the user viewpoint image become mismatched, the image processing device 41 generates an adjusted user viewpoint image, and an AR virtual vehicle is generated based on the adjusted user viewpoint image. In this embodiment, the image processing device 41 generates the adjusted user viewpoint image based on the real vehicle C1, but the adjusted user viewpoint image may also be generated based on any object such as a static object (e.g., a road or a sign).

[0070] As described above, according to this embodiment, an AR virtual vehicle is generated based on an adjusted user viewpoint image generated in response to changes in the height, orientation, and posture of the user 1 wearing the AR glasses 10, and an AR image is generated using the AR virtual vehicle. This allows the AR image to be displayed on the AR glasses 10 in a manner that is more natural to the user 1, allowing the user 1 in the real world to enjoy a more realistic display manner.

[0071] Although the present embodiment describes the generation of AR images when multiple vehicles are moving in the real world and the virtual world, the objects that move in the real world and the virtual world are not limited to vehicles. The present embodiment can be applied to any dynamic or static object whose front-to-back relationship can change between the real world and the virtual world.

[0072] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present disclosure. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present disclosure. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present disclosure.

[0073] (Embodiments of the present disclosure) The present disclosure includes the following embodiments. [1] An augmented reality system comprising one or more processors, wherein at least one of the one or more processors executes a first generation process for generating a virtual viewpoint image from a specified position in a virtual world; a determination process for determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the virtual viewpoint image; a second generation process for generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image; and a display process for displaying the AR object on a transparent display.

[0074] [2] The augmented reality system described in [1], wherein the second generation process includes generating the AR object by removing the area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image.

[0075] [3] The augmented reality system described in [1] or [2], wherein at least one of the one or more processors further executes a third generation process to adjust the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image, wherein the determination process includes determining whether the simulated object and the virtual object overlap in the adjusted virtual viewpoint image, and the second generation process includes generating the AR object according to the positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image.

[0076] [4] The augmented reality system described in [3], wherein in the third generation process, the deviation between the virtual object shown in the adjusted virtual viewpoint image and the object corresponding to the virtual object shown in the real image is smaller than the deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image.

[0077] [5] An augmented reality system described in any of [1] to [4], wherein the real object is a vehicle running in the real world, and the virtual object is an object simulating a vehicle running in the virtual world.

[0078] [6] An image processing device having one or more processors, wherein at least one of the one or more processors executes a determination process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world, and a generation process of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image.

[0079] [7] The image processing device described in [6], wherein the generation process generates the AR object by removing the area where the simulated object and the virtual object overlap from the virtual object when the simulated object is in front and the virtual object is in the back in the virtual viewpoint image.

[0080] [8] The image processing device described in [6] or [7], wherein at least one of the one or more processors further executes an adjustment process for adjusting the virtual viewpoint image using a real image captured by a user device to generate an adjusted virtual viewpoint image, wherein the determination process includes determining whether the simulated object and the virtual object overlap in the adjusted virtual viewpoint image, and the generation process includes generating the AR object according to the positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image.

[0081] [9] The image processing device described in [8], wherein, in the adjustment process, the deviation between the virtual object shown in the adjusted virtual viewpoint image and the object corresponding to the virtual object shown in the real image is smaller than the deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image.

[0082]

[10] A determination step of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world; and a generation step of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image. An image processing method comprising:

[0083]

[11] A computer-readable storage medium storing a program, the program including instructions that, when executed by one or more processors of an image processing device, cause the image processing device to execute a determination process of determining whether a simulated object in the virtual world corresponding to a real object in the real world overlaps with a virtual object that exists in the virtual world and is different from the simulated object in a virtual viewpoint image from a specified position in the virtual world; and a generation process of generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with the positional relationship between the simulated object and the virtual object in the virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the virtual viewpoint image. [Explanation of symbols]

[0084] 1: User, 10: AR glasses, 20: Remote monitoring device, 30: Virtual space providing server, 40; 41: Image processing device, 101: CPU, 102: ROM, 103: RAM, 104: HDD, 105: Input unit, 106: Display unit, 107: Communication unit, 108: Imaging unit, 111: Display control unit, 112: AR glasses position acquisition unit, 113: Imaging control unit, 201: CPU, 202: ROM, 203: RAM, 204, HDD, 205 : Communication unit, 211: Real vehicle management unit, 212: AR glasses position management unit, 213: Vehicle identifier information, 301: CPU, 302: ROM, 303: RAM, 304: HDD, 305: Communication unit, 311: Virtual world management unit, 312: Image generation unit 312, 313: Content information, 401: CPU, 402: ROM, 403: RAM, 404: HDD, 405: Communication unit, 411; 412: Image processing unit, 413: Image adjustment model

Claims

1. a first generation processing unit that generates a virtual viewpoint image from a specified position in the virtual world; a first determination processing unit that determines whether a real object in the real world in a real image captured by a user device matches a color or shape of a simulated object in the virtual world, thereby determining whether the real object and the simulated object are in a correspondence relationship; a second generation processing unit that, when it is determined that the real object and the simulated object are in a corresponding relationship, adjusts the virtual viewpoint image using the real object as a reference to generate an adjusted virtual viewpoint image so that the real object and the simulated object coincide with each other; a second determination processing unit that determines whether or not the simulated object overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the adjusted virtual viewpoint image; a third generation processing unit that generates an AR (augmented reality) object for an AR image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image; a display processing unit that displays the AR object on a transparent display; An augmented reality system having:

2. The third generation processing unit generating the AR object by deleting an overlapping area between the simulated object and the virtual object from the virtual object when the simulated object is in front and the virtual object is in the back in the adjusted virtual viewpoint image; The augmented reality system of claim 1 .

3. a deviation between the virtual object shown in the adjusted virtual viewpoint image and an object corresponding to the virtual object shown in the real image is smaller than a deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image; The augmented reality system of claim 1 .

4. the real object is a vehicle that runs in the real world; The virtual object is an object that resembles a vehicle that runs in the virtual world. The augmented reality system of claim 1 .

5. A first determination processing unit that determines whether a real object in the real world in a real image captured by a user device corresponds to a simulated object in a virtual world by determining whether the color and shape of the real object match; a first generation processing unit that, when it is determined that the real object and the simulated object are in a corresponding relationship, adjusts a virtual viewpoint image from a specified position in the virtual world using the real object as a reference so that the real object and the simulated object coincide with each other, thereby generating an adjusted virtual viewpoint image; a second determination processing unit that determines whether or not the simulated object overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the adjusted virtual viewpoint image; a second generation processing unit that generates an AR (augmented reality) object for an AR image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image; An image processing device comprising:

6. The second generation processing unit generating the AR object by deleting an overlapping area between the simulated object and the virtual object from the virtual object when the simulated object is in front and the virtual object is in the back in the adjusted virtual viewpoint image; The image processing device according to claim 5 .

7. a deviation between the virtual object shown in the adjusted virtual viewpoint image and an object corresponding to the virtual object shown in the real image is smaller than a deviation between the virtual object shown in the virtual viewpoint image and the object corresponding to the virtual object shown in the real image; The image processing device according to claim 5 .

8. An image processing method executed by an image processing device, comprising: determining whether a real object in the real world in a real image captured by the user device matches a color or shape of a simulated object in the virtual world, thereby determining whether the real object and the simulated object correspond to each other; When it is determined that the real object and the simulated object are in a corresponding relationship, using the real object as a reference, adjust a virtual viewpoint image from a specified position in the virtual world so that the real object and the simulated object coincide with each other, thereby generating an adjusted virtual viewpoint image; determining whether or not the simulated object overlaps with a virtual object that exists in the virtual world and is different from the simulated object in the adjusted virtual viewpoint image; when it is determined that the simulated object and the virtual object overlap in the adjusted virtual viewpoint image, generating an AR object for an AR (augmented reality) image corresponding to the virtual object in accordance with a positional relationship between the simulated object and the virtual object in the adjusted virtual viewpoint image; An image processing method comprising:

Images