Display control device

Abstract

In the present invention, an image acquisition unit acquires a first image which has been generated by capturing an image of a prescribed range of a real space at a first time and a second image which has been generated by capturing an image of the prescribed range at a second time that is later than the first time. A detection unit detects the difference between the first image and the second image. A virtual object acquisition unit acquires a virtual object on the basis of the difference detected by the detection unit. A display control unit controls display of the virtual object.

Description

Technical Field

[0001] The present invention relates to a display control device.

Background Art

[0002] Conventionally, XR technologies represented by AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) have been spreading. For example, Patent Document 1 below discloses an AR display management system including a portable terminal device and an authentication device that performs display authentication of virtual objects (AR images). The portable terminal device acquires position information indicating the position of the device and direction information indicating the imaging direction of the imaging unit. The portable terminal device transmits a display authentication request message for a virtual object to the authentication device. The display authentication request message includes an identifier of the virtual object, position information, and direction information. The authentication device authenticates whether to permit the display of the virtual object corresponding to the identifier based on the identifier, position information, and direction information included in the received display authentication request message. The authentication device transmits a response message including the authentication result to the portable terminal device. The portable terminal device determines whether the display of the virtual object is possible based on the received response message. When the portable terminal device determines that the display is possible, it superimposes and displays the virtual object on the captured image displayed on the display unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Virtual objects displayed using XR technology have pre-generated display data. Therefore, depending on the surrounding environment when the virtual object is displayed, it may not blend in with its surroundings, potentially causing the user to feel uneasy.

[0005] The objective of this invention is to display virtual objects in a way that does not create a sense of incongruity with the real world. [Means for solving the problem]

[0006] A display control device according to one aspect of the present invention includes: an image acquisition unit that acquires a first image generated by imaging a predetermined range of real space at a first time point and a second image generated by imaging the predetermined range at a second time point after the first time point; a detection unit that detects the difference between the first image and the second image; a virtual object acquisition unit that acquires a virtual object based on the difference detected by the detection unit; and a display control unit that controls the display of the virtual object. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to display virtual objects in a way that does not create a sense of incongruity with the real world. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the overall configuration of the information processing system 1 according to the embodiment. [Figure 2] The composition of AR glasses 30 is a cylinder. [Figure 3] This is a block diagram showing the configuration of the terminal device 40. [Figure 4] This is a block diagram showing the configuration of server 20. [Figure 5] This figure shows an example of a virtual object database DB1 according to the first embodiment. [Figure 6] This figure shows an example of an image captured by the imaging device 324. [Figure 7] This figure shows an example of an image captured by the imaging device 324. [Figure 8] This figure shows an example of an image captured by the imaging device 324. [Figure 9] This figure shows an example of how virtual objects are displayed. [Figure 10] This figure shows an example of how virtual objects are displayed. [Figure 11] This is a flowchart showing the operation of the processing unit 406. [Figure 12] This figure shows an example of a virtual object database DB2 in the second embodiment. [Figure 13] This figure shows an example of an image captured by the imaging device 324. [Figure 14] This figure shows an example of how virtual objects are displayed. [Figure 15] This figure shows an example of a virtual object database DB3 in the third embodiment. [Figure 16] This figure shows an example of how virtual objects are displayed. [Modes for carrying out the invention]

[0009] A. First Embodiment The configuration of the information processing system 1, which includes a display control device according to the first embodiment of the present invention, will be described below.

[0010] A-1. System Configuration Figure 1 is a block diagram showing the overall configuration of the information processing system 1 according to the embodiment. The information processing system 1 comprises a display system 10 and a server 20.

[0011] In this embodiment, the information processing system 1 is a system that presents various types of information to a user wearing an AR glass 30 described later by means of AR technology. Here, the AR technology is, for example, a technology in which a virtual object is superimposed and displayed on the real space in a device such as a see-through type head-mounted display, so that the virtual object is visually recognized by the user as if it exists in the real space. The virtual object is, for example, a still image, a moving image, a 3DCG model, and text. Note that the AR glass 30 may output other types of information such as voice information when displaying the virtual object.

[0012] The display system 10 and the server 20 are communicably connected to each other via a communication network N. In FIG. 1, only one display system 10 is illustrated, but the information processing system 1 can include an arbitrary number of display systems 10.

[0013] The display system 10 is used when the user visually recognizes a virtual object. The display system 10 includes an AR glass 30 and a terminal device 40. The terminal device 40 is an example of a display control device. The terminal device 40 is communicably connected to the server 20. The terminal device 40 acquires virtual object data VD (see FIG. 4) corresponding to the virtual object output by the AR glass 30 from the server 20. Further, the terminal device 40 causes the AR glass 30 to output a virtual object corresponding to the virtual object data VD acquired from the server 20. The terminal device40 is preferably a portable terminal device such as a smartphone and a tablet.

[0014] Alternatively, the functions of the display system 10 described below may be realized using only the terminal device 40, for example, without providing the AR glasses 30. Specifically, the terminal device 40 includes an imaging device 324, and displays the captured image generated by the imaging device 324 capturing an object on the touch panel 401 (see Figure 3). The terminal device 40 is connected to the server 20 so as to be able to communicate with each other. The terminal device 40 obtains virtual object data VD corresponding to the virtual object to be output by the AR glasses 30 from the server 20. Then, the terminal device 40 displays the virtual object corresponding to the virtual object data VD obtained from the server 20, superimposed on the captured image displayed on the touch panel 401.

[0015] A-2. AR Glasses 30 Figure 2 is a block diagram showing the configuration of the AR glasses 30. The AR glasses 30 are a see-through type head-mounted display worn on the user's head. The shape of the AR glasses 30 is similar to that of ordinary eyeglasses. The AR glasses 30 have a left lens positioned in front of the user's left eye, a right lens positioned in front of the user's right eye, and a frame that supports the left and right lenses. The frame has a bridge between the left and right lenses, and a pair of temples that rest on the left and right ears.

[0016] The AR glasses 30 include a projection device 321, a speaker 322, a communication device 323, an imaging device 324, a GPS device 325, a storage device 327, a processing device 328, and a bus 329. Each of the components shown in Figure 2 is stored, for example, in a frame. The projection device 321, speaker 322, communication device 323, imaging device 324, GPS device 325, storage device 327, and processing device 328 are interconnected by a bus 329 for communicating information. The bus 329 may be configured using a single bus, or different buses may be configured between each element of the device.

[0017] The projection device 321 includes left and right lenses, a display panel, and optical components. The display panel and optical components are housed, for example, in a frame. The display panel and optical components may be provided in pairs, one on the left and one on the right, corresponding to the left and right lenses. Based on control from the terminal device 40, the projection device 321 displays a projected image corresponding to a virtual object on the display panel. The display panel is, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The optical components guide the light emitted from the display panel to the left and right lenses.

[0018] Each of the left and right lenses has a half-mirror. The half-mirrors in the left and right lenses transmit light representing real space, guiding that light to the user's eyes. The half-mirrors in the left and right lenses also reflect light representing virtual objects, guided by the optical components, back towards the user's eyes. When the light from real space transmitted through the half-mirrors and the light representing virtual objects reflected by the half-mirrors are superimposed and incident on the user's eyes, the user perceives that virtual objects are located in real space. In other words, the left and right lenses function as transmissive displays positioned in front of the eyeball.

[0019] Speaker 322 outputs sound. The sound output by speaker 322 is, for example, sound output associated with a virtual object. Speaker 322 is controlled by terminal device 40 or AR glasses 30. Speaker 322 is placed, for example, in the frame. Alternatively, speaker 322 may be separate from AR glasses 30 and not included in them.

[0020] The communication device 323 is equipped with a communication interface for communicating with other devices. The communication device 323 communicates with the terminal device 40 using wireless or wired communication. In this embodiment, the communication device 323 communicates with the communication device 403 (see Figure 3) of the terminal device 40 using short-range wireless communication such as Bluetooth®.

[0021] The imaging device 324 is, for example, a camera. The imaging device 324 has, for example, an imaging optical system and an image sensor. The imaging optical system is an optical system that includes at least one imaging lens. The imaging lens is provided, for example, on the bridge described above. Therefore, the imaging device 324 images the space in the real space that exists in the direction that the user's face is facing. The imaging optical system may have various optical elements such as prisms, or it may have a zoom lens or a focus lens. The image sensor is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0022] The imaging device 324 outputs image information indicating an image generated by imaging a subject (hereinafter referred to as "imaged image"). The imaging range of the imaging device 324 coincides with or includes the user's field of view. In this embodiment, the imaging range of the imaging device 324 is assumed to coincide with the user's field of view. Therefore, the image information indicates the real space within the user's field of view. The correspondence between each pixel of the image captured by the imaging device 324 and each pixel of the transmissive display realized by the left and right lenses is pre-calibrated. That is, it is known which position on the left and right lenses the object in the image occupies as seen by the user wearing the AR glasses 30. Therefore, the display control unit 416, described later, can project a virtual object whose display position has been determined based on the image onto the left and right lenses.

[0023] The captured image generated by the imaging device 324 is transmitted to the terminal device 40 via the communication device 323 as captured image information. The imaging device 324 repeatedly captures images at predetermined intervals, and each time an image is captured, it transmits the generated captured image information to the terminal device 40. Position information indicating the position of the AR glasses 30 at the time of imaging is added to the captured image information.

[0024] The GPS device 325 receives radio waves from multiple satellites and generates location information from the received radio waves. The location information indicates the location of the AR glasses 30. The location information can be in any format as long as it can identify the location. In this embodiment, latitude and longitude are used as location information. Note that the location information may be obtained by means other than the GPS device 325. For example, information transmitted from a beacon installed at a facility may be received as location information. The information transmitted from the beacon etc. is the name of the facility where the AR glasses 30 are located and information that identifies the location within the facility. In this embodiment, the location of the AR glasses 30 is the location of the display system 10. Note that the GPS device 325 may be installed at the terminal device 40. In this case, the GPS device 325 generates location information indicating the location of the terminal device 40. Also, the location of the display system 10 is the location of the terminal device 40.

[0025] The storage device 327 is a recording medium that can be read by the processing device 328. The storage device 327 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The storage device 327 stores program PG1. Program PG1 is a program for operating the AR glasses 30.

[0026] The processing unit 328 includes one or more CPUs (Central Processing Units). One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0027] The processing unit 328 reads program PG1 from the storage device 327. By executing program PG1, the processing unit 328 functions as an operation control unit 330. The operation control unit 330 may be composed of circuits such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array).

[0028] The motion control unit 330 controls the operation of the AR glasses 30. For example, the motion control unit 330 provides the projection device 321 with an image display control signal received by the communication device 323 from the terminal device 40. The projection device 321 displays the image indicated by the image display control signal.

[0029] A-3. Terminal device 40 Figure 3 is a block diagram showing the configuration of the terminal device 40. The terminal device 40 includes a touch panel 401, a communication device 403, a storage device 405, a processing unit 406, and a bus 407. The touch panel 401, the communication device 403, the storage device 405, and the processing unit 406 are interconnected by a bus 407 for communicating information. The bus 407 may be configured using a single bus, or different buses may be configured for each device.

[0030] The touch panel 401 displays various information to the user and detects the user's touch operations. The touch panel 401 serves as both an input and output device. For example, the touch panel 401 is constructed by bonding a touch sensor unit capable of detecting touch operations between a cover glass and various display panels such as a liquid crystal display panel or an organic EL display panel. For example, when the user's finger is in contact with the touch panel 401, the touch panel 401 periodically detects the contact position of the user's finger on the touch panel 401 and transmits touch information indicating the detected contact position to the processing unit 406.

[0031] The communication device 403 is equipped with a communication interface for communicating with other devices. The communication device 403 communicates with the AR glasses 30 using wireless or wired communication. In this embodiment, the communication device 403 communicates with the communication device 323 (see Figure 2) using the same type of short-range wireless communication as the communication device 323 of the AR glasses 30. The communication device 403 also communicates with the server 20 using wireless or wired communication. In this embodiment, the communication device 403 is equipped with an interface that can be connected to the communication network N and communicates with the communication device 203 (see Figure 4) of the server 20 via the communication network N.

[0032] The storage device 405 is a recording medium that can be read by the processing unit 406. The storage device 405 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM, EPROM, and EEPROM. Volatile memory is, for example, RAM. The storage device 405 stores program PG2. Program PG2 is a program for operating the terminal device 40.

[0033] The processing unit 406 includes one or more CPUs. One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0034] The processing unit 406 reads program PG2 from the storage device 405. By executing program PG2, the processing unit 406 functions as an image acquisition unit 410, a detection unit 412, a virtual object acquisition unit 414, and a display control unit 416. At least one of the image acquisition unit 410, the detection unit 412, the virtual object acquisition unit 414, and the display control unit 416 may be configured by a circuit such as a DSP, ASIC, PLD, or FPGA. Details of the image acquisition unit 410, the detection unit 412, the virtual object acquisition unit 414, and the display control unit 416 will be described later.

[0035] A-4. Server 20 Figure 4 is a block diagram showing the configuration of server 20. Server 20 includes a communication device 203, a storage device 205, a processing unit 206, and a bus 207. The communication device 203, the storage device 205, and the processing unit 206 are interconnected by a bus 207 for communicating information. The bus 207 may be configured using a single bus, or different buses may be configured for each device.

[0036] The communication device 203 communicates with the terminal device 40 (see Figures 1 and 3) via wireless or wired communication. In this embodiment, the communication device 203 has an interface that can connect to the communication network N and communicates with the terminal device 40 via the communication network N.

[0037] The storage device 205 is a recording medium that can be read by the processing device 206. The storage device 205 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM, EPROM, and EEPROM. Volatile memory is, for example, RAM. The storage device 205 stores program PG3, multiple virtual object data VD, and virtual object database DB (DB1 to DB3). In Figure 4, the term "database" is abbreviated as "DB". Program PG3 is a program for operating server 20. Virtual object data VD is data for outputting virtual objects with AR glasses 30. Multiple virtual object data VD each correspond to a different virtual object. The virtual object database DB associates differential information, described later, with one virtual object data VD (or two or more virtual object data VD).

[0038] The processing unit 206 includes one or more CPUs. One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0039] The processing unit 206 reads program PG3 from the storage device 205. The processing unit 206 functions as an operation control unit 210 by executing program PG3. The operation control unit 210 may be composed of circuits such as a DSP, ASIC, PLD, and FPGA.

[0040] The operation control unit 210 controls the operation of the server 20. For example, when the operation control unit 210 receives differential information from the terminal device 40, it refers to the virtual object database DB and selects the virtual object data VD associated with the contents of the differential information. The operation control unit 210 also sends the selected virtual object data VD to the terminal device 40.

[0041] A-5. Details of Processing Unit 406 Next, we will describe in detail the image acquisition unit 410, detection unit 412, virtual object acquisition unit 414, and display control unit 416, which are realized when the processing unit 406 executes the program PG2.

[0042] The image acquisition unit 410 acquires a first image generated by imaging a predetermined range of real space at a first time step, and a second image generated by imaging the predetermined range at a second time step after the first time step. In this embodiment, the image acquisition unit 410 acquires image information corresponding to the image generated by the imaging device 324 of the AR glasses 30 at predetermined time intervals via the communication device 403. Each image information is accompanied by location information indicating the location where the imaging to generate the image was performed. In this embodiment, the location information is generated by the GPS device 325. In this embodiment, if the location information attached to the first image generated by imaging at the first time step matches the location information attached to the second image generated by imaging at the second time step after the first time step, it is determined that the first and second images were generated by imaging the same range (predetermined range) of real space.

[0043] Figures 6 to 8 show examples of images generated by the imaging device 324. Image P1 shown in Figure 6 is an image generated by imaging the range R1 in real space at, for example, 2 PM on April 1, 2022. Image P2 shown in Figure 7 is an image generated by imaging the range R1 in real space at, for example, 4 PM on April 1, 2022. That is, image P2 is an image generated by imaging the same location as in image P1, but at a later time than when image P1 was captured. In the comparison between Figure 6 and Figure 7, image P1 is an example of a first image, and image P2 is an example of a second image.

[0044] The detection unit 412 detects the difference between the first image and the second image. In the first embodiment, the difference between the first image and the second image is, for example, the difference in the objects depicted in each image. The detection unit 412 uses image analysis techniques such as semantic segmentation to detect the type (name) of the object depicted in the captured image and the area occupied by that object in the captured image. The area occupied by the object in the captured image corresponds to the position of that object in real space.

[0045] The detection unit 412 detects whether the first object, which is visible in the second image, is also visible in the first image. If the first object, which is visible in the second image, is not visible in the first image, there is a difference between the first and second images. That is, it is presumed that the first object moved from another location into the area visible in the second image (a predetermined area) between the first and second time points. Also, if the first object, which is visible in the second image, is also visible in the first image, there is no difference between the first and second images. That is, it is presumed that the first object remained within the area visible in both the first and second images between the first and second time points.

[0046] In the first embodiment, the difference between the position of the first object in the first image and the position of the first object in the second image is not considered. That is, if the first object is visible in both the first and second images, but its position differs between the two images, the first and second images are treated as if there is no difference.

[0047] Comparing image P1 and image P2, image P2 shows bicycle B and car C parked on road S in front of house H, but image P1 does not show car C. Therefore, there is a difference between image P1 and image P2 in the presence or absence of car C. More specifically, when viewing image P1 and image P2 in accordance with the time of acquisition, it can be seen that car C newly appeared in range R1. In this case, the detection unit 412 detects that "car" has become "+".

[0048] Furthermore, the detection unit 412 detects whether the second object, which is visible in the first image, is also visible in the second image. If the second object is visible in the first image but not in the second image, there is a difference between the first and second images. That is, it is presumed that the second object moved to another location outside the area visible in the first image between the first and second time points. Conversely, if the second object is visible in the first image but also in the second image, there is no difference between the first and second images. That is, it is presumed that the second object remained within the area visible in both the first and second images between the first and second time points.

[0049] Image P3, shown in Figure 8, is an image generated by capturing the real-world area R1 at, for example, 18:00 on April 1, 2022. In other words, image P3 is an image generated by capturing the same location as in image P2 (see Figure 7) but at a later time than when image P2 was captured. In the comparison between Figure 7 and Figure 8, image P2 is an example of the first image, and image P3 is an example of the second image.

[0050] Comparing image P2 and image P3, image P2 shows bicycle B and car C parked on road S in front of house H, but image P3 does not show bicycle B. Therefore, there is a difference between image P2 and image P3 in the presence or absence of bicycle B. More specifically, when looking at image P2 and image P3 in accordance with the time of acquisition, it can be seen that bicycle B moved outside the range R1. In this case, the detection unit 412 detects that "bicycle" has become "-".

[0051] The virtual object acquisition unit 414 acquires virtual objects based on the differences detected by the detection unit 412. Before describing the virtual object acquisition unit 414, the virtual object database DB (see Figure 4) owned by the server 20 will be described.

[0052] Figure 5 shows an example of a virtual object database DB1 according to the first embodiment. The virtual object database DB1 stores records R11 to R14, each containing an object name M11, an occurrence condition M12, and a virtual object data name M13. The object name M11 indicates the name of an object that exists in real space. In Figure 5, a broad classification of the object, such as "automobile," is used as the object name M11, but for example, a classification of automobiles (bus, truck, or private car, etc.) or a vehicle type may be used as the object name M11.

[0053] Appearance condition M12 indicates the change in the appearance of an object indicated by object name M11 in two images (for example, the first image and the second image) generated by imaging the same location at different times. Specifically, appearance condition M12 is indicated by "+" or "-". When comparing the first image generated by imaging at the first time point with the second image generated by imaging at the second time point (later than the first time point), if there is an object that appears only in the second image, the appearance condition M12 for that object will be "+". In other words, when viewing the first and second images in chronological order of imaging, an object that newly appears will have an appearance condition M12 of "+". Also, when comparing the first and second images, if there is an object that appears only in the first image, the appearance condition M12 for that object will be "-". In other words, when viewing the first and second images in chronological order of imaging, an object that has disappeared from its location will have an appearance condition M12 of "-".

[0054] The virtual object data name M13 is identification information that identifies the virtual object data VD (see Figure 4). For example, record R11 indicates that when the occurrence condition for "car" becomes "+", the virtual object data VD with the data name "aaaa.xxx" is selected. Similarly, record R12 indicates that when the occurrence condition for "car" becomes "-", the virtual object data VD with the data name "bbbb.xxx" is selected.

[0055] The virtual object acquisition unit 414 acquires virtual object data VD from the server 20. More specifically, if the virtual object acquisition unit 414 detects a difference in the presence or absence of an object as a result of comparing the first image and the second image, it sends difference information to the server 20, including the object name and the conditions for the appearance of the object. The operation control unit 210 of the server 20 queries the virtual object database DB using the object name and appearance conditions included in the difference information and identifies the virtual object data name M13. The operation control unit 210 of the server 20 selects the virtual object data VD corresponding to the virtual object data name M34 from among the multiple virtual object data VD stored in the storage device 205. The selected virtual object data VD is sent to the terminal device 40. Through the above process, the virtual object acquisition unit 414 acquires the virtual object data VD.

[0056] For example, if image P1 shown in Figure 6 is the first image and image P2 shown in Figure 7 is the second image, the virtual object acquisition unit 414 sends difference information to the server 20 with the object name set to "automobile" and the appearance condition set to "+". The operation control unit 210 of the server 20 queries the virtual object database DB and selects the virtual object data VD that matches the difference information. The virtual object data VD corresponding to the object name "automobile" and the appearance condition "+" is "aaaa.xxx" as described in record R11. The operation control unit 210 of the server 20 sends the virtual object data VD with the data name "aaaa.xxx" to the terminal device 40. The virtual object acquisition unit 414 acquires the virtual object data VD with the data name "aaaa.xxx" as the virtual object data VD corresponding to "automobile". In other words, if the automobile C shown in image P2 is not shown in image P1, the virtual object acquisition unit 414 acquires the virtual object data VD corresponding to the virtual object associated with automobile C.

[0057] For example, if image P2 shown in Figure 7 is the first image and image P3 shown in Figure 8 is the second image, the virtual object acquisition unit 414 sends difference information to the server 20 with the object name set to "bicycle" and the appearance condition set to "-". The operation control unit 210 of the server 20 queries the virtual object database DB and selects the virtual object data VD that matches the difference information. The virtual object data VD corresponding to the object name "bicycle" and the appearance condition "-" is "dddd.xxx" as described in record R14. The operation control unit 210 of the server 20 sends the virtual object data VD with the data name "dddd.xxx" to the terminal device 40. The virtual object acquisition unit 414 acquires the virtual object data VD with the data name "dddd.xxx" as the virtual object data VD corresponding to "bicycle". In other words, if the bicycle shown in image P2 is not shown in image P3, the virtual object acquisition unit 414 acquires the virtual object data VD associated with bicycle B.

[0058] In this embodiment, the appearance condition M12 in the virtual object database DB was indicated as "+ (newly appearing)" or "- (disappearing)," and a virtual object data name M13 was associated with each. Alternatively, for example, the virtual object data name M13 could be associated with the appearance condition M12 only when the appearance condition M12 is "+", or the virtual object data name M13 could be associated with the appearance condition M12 only when the appearance condition M12 is "-". Furthermore, for example, the appearance condition M12 could be set to indicate that an object appears in both the first and second images, that is, that the appearance status of the object does not change (for example, it could be indicated as "0"), and the virtual object data name M13 could be associated with this condition.

[0059] Furthermore, for example, a virtual object database DB1 may be stored in the terminal device 40. Since there are countless objects in real space, it is not possible to associate a virtual object with every object. By storing the virtual object database DB1 on the terminal device 40, the processing unit 406 only needs to extract the difference of the object stored in the object name M11 of the virtual object database DB1 from the captured image, thereby reducing the processing load. In this case, the processing unit 406 identifies the virtual object data name M13 corresponding to the difference of the object that occurred in real space, based on the virtual object database DB1. Then, the processing unit 406 requests the server 20 to send the virtual object data VD identified by the virtual object data name M13.

[0060] The display control unit 416 controls the display of virtual objects. In this embodiment, the display control unit 416 uses the virtual object data VD acquired by the virtual object acquisition unit 414 to control the projection device 321 of the AR glasses 30 and display the virtual object corresponding to the virtual object data VD on the AR glasses 30.

[0061] Figures 9 and 10 show examples of how virtual objects are displayed. Figure 9 schematically shows the field of view U1 of a user wearing AR glasses 30 at approximately the same time as the image acquisition time of image P2 (16:00 on April 1, 2022). As described above, the imaging range of the imaging device 324 matches the user's field of view, so the field of view U1 is approximately the same as that of image P2 shown in Figure 7. On the other hand, the field of view U1 includes a virtual object V1 displayed on the AR glasses 30 by the display control unit 416. The virtual object V1 is a virtual object corresponding to the virtual object data VD with the data name "aaaa.xxx". The virtual object V1 is displayed near the car C, which is the difference between image P1 and image P2 (on the roof of car C in the example of Figure 9). The display position of the virtual object V1 may be set in the virtual object data VD.

[0062] Figure 10 schematically shows the field of view U2 of a user wearing AR glasses 30 at approximately the same time as the acquisition time of image P3 (18:00 on April 1, 2022). The field of view U2 is approximately the same as that of image P3 shown in Figure 8. On the other hand, the field of view U2 includes a virtual object V2 displayed on the AR glasses 30 by the display control unit 416. The virtual object V2 is a virtual object corresponding to the virtual object data VD with the data name "dddd.xxx". The virtual object V2 is displayed near the location where bicycle B is located, which is the difference between image P2 and image P3.

[0063] Figure 11 is a flowchart showing the operation of the processing unit 406. The processing unit 406 functions as an image acquisition unit 410 and acquires an image from the imaging device 324 of the AR glasses 30 (step S101). The processing unit 406 functions as a detection unit 412 and determines whether there is an image among previously acquired images whose position information matches the position information attached to the image acquired in step S101 (hereinafter referred to as a "corresponding image") (step S102). If there is no corresponding image (step S102: NO), the processing unit 406 repeats the process of step S101.

[0064] If a corresponding image exists (step S102: YES), the processing unit 406 functions as a detection unit 412 and compares the image acquired in step S101 with the corresponding image (step S103). If there are multiple corresponding images, the processing unit 406 selects, for example, the image whose acquisition time is closest to the acquisition time of the image acquired in step S101 as the corresponding image.

[0065] If there is a difference between the two captured images (step S104: YES), the processing unit 406 functions as a virtual object acquisition unit 414 and sends the difference information to the server 20 (step S105). The processing unit 406 waits until it receives virtual object data VD from the server 20 (step S106: NO).

[0066] When the virtual object data VD is received from the server 20 (step S106: YES), the processing unit 406 functions as a display control unit 416 and displays the virtual object corresponding to the virtual object data VD on the AR glasses 30 (step S107). The display of the virtual object may end, for example, when the user's field of view no longer includes the display position of the virtual object. Alternatively, depending on the content of the virtual object, the display position of the virtual object may be moved so that the virtual object is continuously included in the user's field of view. Also, if there is no difference between the two captured images in step S104 (step S104: NO), the processing unit 406 returns to step S101.

[0067] A-4. Summary of the First Embodiment As described above, the display system 10 according to the first embodiment compares two captured images generated by imaging the same location, and if there is a change in the object in the captured image, it acquires a virtual object associated with that object and displays the virtual object on the AR glasses 30. Therefore, the display system 10 displays a virtual object that reflects the change in the presence or absence of the object in the real space, and can display a virtual object that does not feel out of place in the real space.

[0068] Furthermore, the display system 10 according to the first embodiment generates a second image by capturing the same location as the first image after capturing the first image. If an object in the second image is not captured in the first image, the display system 10 acquires a virtual object associated with that object and displays the virtual object on the AR glasses 30. In other words, when a new object appears in the real world, the display system 10 displays a virtual object corresponding to that object. Therefore, it is possible to draw the user's attention to objects that appear in the real world. By displaying such virtual objects, it is possible to improve safety when the user passes around an object, to advertise the object more effectively, or to change the scenery that the user sees on a daily basis.

[0069] Furthermore, the display system 10 according to the first embodiment acquires a virtual object associated with an object if the object shown in the first image is not shown in the second image, and displays the virtual object on the AR glasses 30. In other words, the display system 10 displays a virtual object corresponding to an object that was previously present in the real world but has disappeared. Thus, it is possible to draw the user's attention to an object that has disappeared from the real world. By displaying such virtual objects, it is possible to, for example, draw the user's attention to changes in their surroundings or to alter the scenery that the user sees on a daily basis.

[0070] B. Second Embodiment A second embodiment of the present invention will be described below. In the following description, for the sake of simplicity, the same reference numerals will be used for components identical to those in the first embodiment, and their functions may be omitted. Also, in the following description, for the sake of simplicity, the differences between the second embodiment and the first embodiment will be described primarily.

[0071] In the first embodiment, the display system 10 displayed a virtual object when there was a difference in the presence or absence of an object between the first image and the second image. In the second embodiment, the display system 10 displayed a virtual object when there was a difference in the state of the same object between the first image and the second image.

[0072] Figure 12 shows an example of a virtual object database DB2 in the second embodiment. The virtual object database DB stores records R21 to R24, which include an object name M21, a part name M22, a state condition M23, and a virtual object data name M24. The object name M21 indicates the name of an object that exists in real space. In Figure 12, a broad classification of the object, such as "tree," is used as the object name M21, but for example, a classification of the tree (coniferous or broad-leaved tree, etc.) or a variety may be used as the object name M21. The part name M22 identifies a part of the object. In Figure 12, the tree and its parts, "leaves" and "flowers," are listed as part names M22.

[0073] State condition M23 indicates the change in the state of the portion indicated by portion name M22 in two images (for example, the first image and the second image) generated by imaging the same location at different times. In this embodiment, state condition M23 is indicated by "+" or "-". When comparing the first image generated by imaging at the first time point with the second image generated by imaging at the second time point, which is later than the first time point, if the amount or number of the portion indicated by portion name M22 has increased, the state condition M23 for that portion will be "+". In other words, when viewing the first image and the second image along the imaging timeline, any portion whose amount or number has increased will have a state condition M23 of "+". Also, when comparing the first image and the second image, if the amount or number of the portion indicated by portion name M22 has decreased, the state condition M23 for that portion will be "-". In other words, when viewing the first image and the second image along the imaging timeline, any portion whose amount or number has decreased will have a state condition M23 of "-". Note that state condition M23 is not limited to increases or decreases in amount or number. For example, the change in the position of an object within an image may be set as state condition M23.

[0074] The virtual object data name M24 is identification information that identifies the virtual object data VD (see Figure 4). For example, record R21 indicates that when the state condition of the "leaf" of the "tree" becomes "+", the virtual object data VD with the data name "eeee.xxx" is selected.

[0075] The image acquisition unit 410 acquires, similar to the first embodiment, a first image generated by imaging a predetermined range of real space at a first time step, and a second image generated by imaging a predetermined range at a second time step that is after the first time step.

[0076] In the second embodiment, the detection unit 412 detects the difference between the state of the third object in the first image and the state of the third object in the second image as the difference between the first image and the second image. If the third object is, for example, a tree, the detection unit 412 compares the state of the tree in the first image with the state of the tree in the second image and detects the difference. The state of the tree is, for example, as shown in the virtual object database DB2 in Figure 12, an increase or decrease in leaves or flowers, etc.

[0077] In the second embodiment, if there is a difference in the state of the third object, the virtual object acquisition unit 414 acquires a virtual object associated with the state of the third object as seen in the second image. If the virtual object acquisition unit 414 detects a difference in the state of the object as a result of comparing the first image and the second image, it transmits difference information to the server 20, which includes the object name of the object (corresponding to the object name M21 in Figure 12) and information that identifies the change in state. In this embodiment, the information that identifies the change in state is a part name that identifies a part of the object (corresponding to the part name M22 in Figure 12) and a "+" or "-" (corresponding to the state condition M23 in Figure 12) that indicates an increase or decrease in the amount or number of the part identified by the part name.

[0078] The operation control unit 210 of the server 20 queries the virtual object database DB using the object name, part name, and state conditions included in the differential information to identify the virtual object data name M24. Then, the operation control unit 210 of the server 20 selects the virtual object data VD corresponding to the virtual object data name M24 from among the multiple virtual object data VDs stored in the storage device 205, and transmits the selected virtual object data VD to the terminal device 40.

[0079] The display control unit 416, similar to the first embodiment, displays the virtual object on the AR glasses 30 by superimposing it onto a predetermined range of the real space (the range captured in the second image).

[0080] Figure 13 shows an example of an image captured by the imaging device 324. Image P4 shown in Figure 13 is an image generated by capturing the range R1 in real space at 2:00 PM on November 1, 2022. In other words, image P4 is an image generated by capturing the same location as in image P1 (see Figure 6) at a later time than when image P1 was captured. In the comparison between Figure 6 and Figure 13, image P1 is an example of the first image, and image P4 is an example of the second image.

[0081] Comparing image P1 and image P4, in image P1, the leaves of tree T at house H are lush, but in image P4, the leaves of tree T at house H are gone. Therefore, there is a difference between image P1 and image P4 in the presence or absence of leaves on the tree. In this case, the detection unit 412 detects that the "leaves" of the "tree" have become "-".

[0082] The virtual object acquisition unit 414 sends difference information to the server 20, with the object name set to "tree", the part name to "leaf", and the state condition set to "-". The operation control unit 210 of the server 20 queries the virtual object database DB shown in Figure 12 and selects the virtual object data VD that matches the difference information. The virtual object data VD corresponding to the object name "tree", part name "leaf", and state condition "-" is "ffff.xxx" as described in record R22. The operation control unit 210 of the server 20 sends the virtual object data VD with the data name "ffff.xxx" to the terminal device 40. The virtual object acquisition unit 414 acquires the virtual object data VD with the data name "ffff.xxx" as the virtual object data VD corresponding to the change in the state of the "tree". That is, if there is a difference in the state of the tree, the virtual object acquisition unit 414 acquires the virtual object associated with the state of the tree shown in image P4 (the leaves are gone).

[0083] Figure 14 shows an example of how a virtual object is displayed. Figure 14 schematically shows the field of view U3 of a user wearing AR glasses 30 at approximately the same time as the acquisition time of image P4 (14:00 on November 1, 2022). The field of view U3 includes a virtual object V3 displayed on the AR glasses 30 by the display control unit 416. The virtual object V3 is a virtual object corresponding to the virtual object data VD with the data name "ffff.xxx". The virtual object V3 is displayed near the tree T, which is the difference between image P1 and image P4 (on the branches of the tree in the example of Figure 14).

[0084] As described above, the display system 10 according to the second embodiment compares two captured images generated by imaging the same location, and if there is a change in the state of an object in the captured image, it acquires a virtual object corresponding to the changed state of the object and displays the virtual object on the AR glasses 30. Therefore, the display system 10 displays a virtual object that reflects the change in the state of the object in the real space, and can display a virtual object that does not feel out of place in the real space.

[0085] C. Third Embodiment A third embodiment of the present invention will be described below. In the following description, for the sake of simplicity, the same reference numerals will be used for components identical to those in the first embodiment, and their functions may be omitted. Also, in the following description, for the sake of simplicity, the differences between the third embodiment and the first embodiment will be described primarily.

[0086] In the first embodiment, the display system 10 displayed a virtual object when there was a difference in the presence or absence of an object between the first image and the second image. In the third embodiment, the display system 10 estimated a change in the environment from the difference between the first image and the second image and displayed a virtual object corresponding to the change in the environment.

[0087] Figure 15 shows an example of a virtual object database DB3 in the third embodiment. The virtual object database DB3 stores records R31 to R34, each containing a region name M31, a target name M32, a state condition M33, and a virtual object data name M34. The region name M31 indicates the name of a region that exists in real space. In Figure 15, "sky" is used as an example of the region name M31. The target name M32 specifies the comparison target from among the regions identified by the region name M31. In Figure 15, "brightness" and "cloud cover" of the sky are listed as target names M32.

[0088] State condition M33 indicates the change in the state of the object indicated by object name M32 in two images (for example, the first image and the second image) generated by imaging the same location at different times. In this embodiment, state condition M33 is indicated by "+" or "-". When comparing the first image generated by imaging at the first time point with the second image generated by imaging at the second time point, which is later than the first time point, if the quantity, number, or degree of the object indicated by object name M32 has increased, the state condition M33 of that object will be "+". Conversely, when comparing the first image and the second image, if the quantity, number, or degree of the object indicated by object name M32 has decreased, the state condition M33 of that object will be "-".

[0089] The virtual object data name M34 is identification information that identifies the virtual object data VD (see Figure 4). For example, record R31 indicates that when the "brightness" state condition of "empty" becomes "+", the virtual object data VD with data name "iiii.xxx" is selected.

[0090] The image acquisition unit 410 acquires, similar to the first embodiment, a first image generated by imaging a predetermined range of real space at a first time step, and a second image generated by imaging a predetermined range at a second time step that is after the first time step.

[0091] In the third embodiment, the detection unit 412 detects changes in the environment within a predetermined range based on the difference between the first image and the second image. For example, the detection unit 412 compares the brightness of the sky in the first image with the brightness of the sky in the second image and detects the difference. Generally, changes in the brightness of the sky are caused by changes in the position of the sun. More specifically, the night sky is nearly black, and the sky becomes brighter as sunrise approaches. Around the time of solar noon, the brightness of the sky begins to decrease, and after sunset it becomes nearly black. If there is a change in the brightness of the sky between the first image and the second image, the detection unit 412 detects whether the change in brightness is "+" or "-".

[0092] In the third embodiment, the virtual object acquisition unit 414 acquires virtual objects associated with environmental changes. If the virtual object acquisition unit 414 detects an environmental change as a result of comparing the first image and the second image, it sends difference information to the server 20 that includes the name of the region where the change is occurring (corresponding to the region name M31 in Figure 15), the name of the object where the change is occurring in that region (corresponding to the object name M32 in Figure 15), and a "+" or "-" (corresponding to the state condition M33 in Figure 15) indicating an increase or decrease in the quantity, number, or degree of the object identified by the object name.

[0093] The operation control unit 210 of the server 20 identifies the virtual object data name M24 based on the differential information, similar to the first embodiment, and transmits the virtual object data VD corresponding to the virtual object data name M24 to the terminal device 40.

[0094] The display control unit 416, similar to the first embodiment, displays the virtual object on the AR glasses 30 by superimposing it onto a predetermined range of the real space (the range captured in the second image).

[0095] For example, let's consider image P1 shown in Figure 6 as the first image and image P3 shown in Figure 8 as the second image. Comparing image P1 and image P3, the sky is bright in image P1, but in image P3, sunset is approaching and the sky is dark. In this case, the detection unit 412 detects that the target name "brightness" for the region name "sky" has become "-".

[0096] The virtual object acquisition unit 414 sends differential information to the server 20 with the area name set to "empty", the target name to "brightness", and the state condition set to "-". The operation control unit 210 of the server 20 queries the virtual object database DB3 shown in Figure 15 and selects the virtual object data VD that matches the differential information. The virtual object data VD corresponding to the area name "empty", the target name "brightness", and the state condition "-" is "jjjj.xxx" as described in record R32. The operation control unit 210 of the server 20 sends the virtual object data VD with the data name "jjjj.xxx" to the terminal device 40. The virtual object acquisition unit 414 acquires the virtual object data VD with the data name "jjjj.xxx" as the virtual object data VD corresponding to the change in the "brightness" state of "empty".

[0097] Figure 16 shows an example of how a virtual object is displayed. Figure 16 schematically shows the field of view U4 of a user wearing AR glasses 30 at approximately the same time as the image P3 was captured (18:00 on April 1, 2022). The field of view U4 includes a virtual object V4 displayed on the AR glasses 30 by the display control unit 416. The virtual object V4 is a virtual object corresponding to the virtual object data VD with the data name "jjjj.xxx".

[0098] As described above, the display system 10 according to the third embodiment compares two captured images generated by imaging the same location, and if there is a change in the environment within the range captured in the images, it acquires a virtual object corresponding to the change in the environment and displays the virtual object on the AR glasses 30. Therefore, the display system 10 can display a virtual object that reflects the change in the environment in the real space, and can display a virtual object that does not feel out of place in the real space.

[0099] C: Variant The following are examples of modifications in the above-described embodiment. Two or more modifications selected from the following examples may be combined as appropriate, provided they do not contradict each other.

[0100] C1: First variation In the first to third embodiments, the AR glasses 30 and the terminal device 40 were separate components. Alternatively, for example, the AR glasses 30 may have the functions of the terminal device 40. That is, the processing unit 328 of the AR glasses 30 may function as an image acquisition unit 410, a detection unit 412, a virtual object acquisition unit 414, and a display control unit 416.

[0101] According to the first modification, the terminal device 40 becomes unnecessary, which is advantageous in simplifying the system configuration.

[0102] C2: Second variation In the first to third embodiments, it was determined whether the two captured images were generated by capturing the same area in real space, based on the position information detected by the GPS device 325. Instead of the GPS device 325, or in addition to the GPS device 325, sensors such as a geomagnetic sensor, acceleration sensor, angular acceleration sensor, or inertial measurement unit (IMU) may be used to detect the orientation of the AR glasses 30 and determine whether the imaging ranges of the two captured images coincide.

[0103] According to the second modification, it is possible to determine with greater accuracy whether the two captured images were generated by capturing the same area in real space, and to display virtual objects that are more adapted to changes in real space.

[0104] D: Other (1) Each function illustrated in Figures 2 to 4 can be implemented by any combination of hardware and software. The method of implementing each function is not particularly limited. Each function may be implemented using a single device that is physically or logically coupled, or it may be implemented using a device that is configured by directly or indirectly connecting two or more physically or logically separated devices (e.g., using wired, wireless, etc.). Each function may be implemented by combining the above single device or the above multiple devices with software.

[0105] (2) In this specification, the term “apparatus” may be replaced with other terms such as circuit, device or unit.

[0106] (3) In each of the first to third embodiments and the first to second modified examples, the storage device 205, storage device 327, and storage device 405 may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.

[0107] (4) Each of the first to third embodiments and the first to second modified examples is a LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or decimal), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE This may apply to at least one system utilizing 802.20, UWB (Ultra-WideBand), Bluetooth®, or other appropriate systems, and to next-generation systems extended, modified, created, or defined based thereon. It may also apply to a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0108] (5) The processing procedures, sequences, flowcharts, etc., illustrated in each of the First to Third Embodiments and the First to Second Modifications may be rearranged in order, as long as they do not contradict each other. For example, the methods described herein present various step elements in an illustrated order and are not limited to the specific order presented.

[0109] (6) In each of the first to third embodiments and the first to second modified examples, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0110] (7) In each of the first to third embodiments and the first to second modified examples, the determination may be based on a value represented by one bit (0 or 1), on a boolean value (true or false), or on a numerical comparison (for example, a comparison with a predetermined value).

[0111] (8) The programs illustrated in each of the First to Third Embodiments and the First to Second Modifications should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures or functions, whether they are called software, firmware, middleware, microcode or hardware description languages ​​or by other names. Furthermore, software, or instructions, etc., may be transmitted or received via a transmission medium. For example, if software is transmitted from a website, server or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair and digital subscriber lines (DSL)) and wireless technologies (such as infrared and microwave), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0112] (9) The information described in each of the First to Third Embodiments and the First to Second Modifications may be represented using any of the following different technologies. For example, the data and information that may be mentioned throughout the above description may be represented in terms of voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. The terms described herein and the terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0113] (10) In each of the First to Third Embodiments and the First to Second Modifications, the terms “system” and “network” are used interchangeably.

[0114] (11) In each of the first to third embodiments and the first to second modifications, the terminal device 40 may be a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.

[0115] (12) A mobile station may also be called a transmitting device, receiving device, or communication device. A mobile station may also be a device mounted on a mobile body, or the mobile body itself. A mobile body means a movable object. The speed of movement of a mobile body is arbitrary. A mobile body is stoppable. A mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on them. A mobile body may be a mobile body that moves autonomously based on operational commands. A mobile body may be a vehicle (e.g., a car, an airplane), an unmanned mobile body (e.g., a drone, an autonomous vehicle), or a robot (manned or unmanned). A mobile station may also include devices that do not necessarily move during communication operations. For example, the mobile station could be an IoT (Internet of Things) device such as a sensor.

[0116] (13) In each of the First to Third Embodiments and the First to Second Modifications, the term “determining” or “deciding” may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database or another data structure), ascertaining, etc. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. Furthermore, "decision" can include considering something as having "decided" to have resolved, selected, chosen, established, compared, etc. In other words, "decision" can include considering some action as having been "decided". Also, "decision" can be rephrased as "assuming", "expecting", or "considering".

[0117] (14) In each of the First to Third Embodiments, and the First to Second Modifications, the term “connected,” or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in the present disclosure, two elements may be considered to be “connected” or “coupled” to each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0118] (15) In each of the First to Third Embodiments and the First to Second Modifications, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on".

[0119] (16) Any reference to elements using designations such as “first” and “second” as used herein does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, references to first and second elements do not imply that only two elements may be adopted or that the first element must precede the second element in any way.

[0120] (17) Where “include,” “including,” and variations thereof are used in each of the First to Third Embodiments and the First to Second Modifications, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, where the term “or” is used in this specification or the claims, it is not intended to be an exclusive OR.

[0121] (18) Where articles are added by translation throughout the Application, for example, a, an, and the in English, the Disclosure may include the fact that the nouns following these articles are plural.

[0122] (19) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Accordingly, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined. [Explanation of Symbols]

[0123] 1... Information processing system, 10... Display system, 20... Server, 30... AR glasses, 40... Terminal device, 410... Image acquisition unit, 412... Detection unit, 414... Virtual object acquisition unit, 416... Display control unit, N... Communication network.

Claims

1. An image acquisition unit that acquires a first image generated by imaging a predetermined range of real space at a first time point, and a second image generated by imaging the predetermined range at a second time point after the first time point. A detection unit for detecting the difference between the first image and the second image, A virtual object acquisition unit acquires a virtual object based on the difference detected by the detection unit, The system comprises a display control unit that controls the display of the virtual object, The detection unit detects the difference between the quantity or number of the first object shown in the first image and the quantity or number of the first object shown in the second image. The virtual object acquisition unit acquires a first virtual object corresponding to the increase in the first portion when the amount or number of the first portion of the first object shown in the second image increases compared to the amount or number of the first portion shown in the first image, and acquires a second virtual object different from the first virtual object corresponding to the decrease in the first portion when the amount or number of the first portion of the first object shown in the second image decreases compared to the amount or number of the first portion shown in the first image. Display control device.

2. The amount or number of the first portion shown in the second image increases compared to the amount or number of the first portion shown in the first image, meaning that the first portion is not shown in the first image but is shown in the second image. The amount or number of the first part shown in the second image is reduced compared to the amount or number of the first part shown in the first image, meaning that the first part is shown in the first image and not shown in the second image. The display control device according to claim 1.

3. The aforementioned virtual object is an image representing a character or a bird. The display control device according to claim 1.

Images