Display Control Device

Abstract

A display control device 10 according to an embodiment of the present invention comprises: a mode detection unit 15 that detects a mode corresponding to the degree of comprehension of the spacial structure of an actual space displayed on a display unit 12; and a control unit 16 that controls the display state of a virtual object in the display unit 12 in accordance with the mode detected by the mode detection unit 15. When the mode detected by the mode detection unit 15 changes from a first mode corresponding to a first degree of comprehension to a second mode corresponding to a second degree of comprehension, the control unit 16 changes the display state of the virtual object in the display unit 12 from a first display state corresponding to the first mode to a second display state corresponding to the second mode via one or more intermediate display states between the first display state and the second display state.

Description

[Technical field]

[0001] One aspect of the present invention relates to a display control device. [Background technology]

[0002] Conventionally, a technology is known that provides a user with a mixed reality (MR) that combines a real space and a virtual space. For example, Patent Document 1 discloses a method of calculating a positional relationship between an object in a real space (hereinafter referred to as a "real object") and an object in a virtual space (hereinafter referred to as a "virtual object") displayed on a display unit such as a display, creating a mask corresponding to the shape of a part of the real object that covers the virtual object based on the calculation result, and superimposing the mask on the virtual space to remove a part of the virtual object that is not displayed (i.e., a part covered by the real object), thereby obtaining an image in which the front-back relationship between the real object and the virtual object is taken into consideration (i.e., an image in which occlusion processing has been performed on the virtual object). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-157606 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to perform the above-mentioned occlusion process appropriately and immediately, it is necessary to acquire data (hereinafter referred to as "spatial data") related to the spatial structure (three-dimensional structure) of the real space displayed on the display unit in advance. However, it is not realistic to acquire spatial data for all locations in the real space in advance. For this reason, it is normal that there are areas in the real space for which spatial data has been acquired (hereinafter referred to as "first areas") and areas for which spatial data has not been acquired (hereinafter referred to as "second areas"). When the first area is displayed on the display unit, the above-mentioned occlusion process can be performed, but when the second area is displayed on the display unit, the above-mentioned occlusion process cannot be performed. For this reason, if the occlusion process is performed when the first area is displayed on the display unit and the occlusion process is not performed when the second area is displayed on the display unit, when the real space displayed on the display unit changes from one of the first area and the second area to the other, the display mode of the virtual object on the display unit changes suddenly, which may cause a sense of incongruity to the user.

[0005] Therefore, an object of one aspect of the present invention is to provide a display control device that can reduce the sense of discomfort felt by a user in a mechanism for presenting a virtual object to a user along with a real space. [Means for solving the problem]

[0006] A display control device according to one aspect of the present invention is a display control device that performs display control of a display unit on which a virtual object is displayed together with a real space. The display control device includes a mode detection unit that detects a mode corresponding to a graspability of a spatial structure of the real space displayed on the display unit, and a control unit that controls a display mode of a virtual object on the display unit according to the mode detected by the mode detection unit. When the mode detected by the mode detection unit changes from a first mode corresponding to a first graspability to a second mode corresponding to a second graspability, the control unit changes the display mode of the virtual object on the display unit from a first display mode corresponding to the first mode to a second display mode corresponding to the second mode by passing through one or more intermediate display modes between the first display mode and the second display mode.

[0007] In a display control device according to one aspect of the present invention, a mode detection unit detects a mode corresponding to a degree of understanding of the spatial structure of the real space displayed on the display unit. Then, a control unit displays a virtual object on the display unit in a display mode corresponding to the mode. More specifically, when the first mode is changed to the second mode, the control unit changes the display mode of the virtual object from a display mode (first display mode) corresponding to the first mode to a display mode (second display mode) corresponding to the second mode via one or more intermediate display modes. That is, when the mode is changed, the control unit does not immediately reflect the display mode (second display mode) corresponding to the latest detected mode (second mode) on the virtual object, but changes the display mode (first display mode) corresponding to the mode before the change (first mode) to a display mode (second display mode) corresponding to the mode after the change (second mode) in a stepwise manner. By the above process, it is possible to prevent the display mode of the virtual object from changing suddenly when the mode is changed. As a result, it is possible to reduce the sense of discomfort of the user. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to provide a display control device that can reduce a sense of discomfort felt by a user in a mechanism for presenting a virtual object to a user together with a real space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a display control device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a content image provided to a user by a display control device. [Diagram 3] 4 is a flowchart showing an example of an operation of the display control device. [Figure 4] FIG. 4 is a diagram illustrating an example of the process in step S5 of FIG. 3. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or corresponding elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0011] 1 is a diagram illustrating an example of a display control device 10 according to an embodiment. The display control device 10 is a computer system that provides a user with an experience of augmented reality (AR) or mixed reality (MR) by providing the user with an image (hereinafter, referred to as a "content image") in which a virtual object (virtual content) is displayed together with a real space.

[0012] The form of the display control device 10 is not limited to a specific form. As an example, the display control device 10 may be a wearable device that is worn on the head of a user. Examples of such a display control device 10 include glasses-type devices (for example, smart glasses such as so-called AR / MR glasses), goggle-type devices, hat-type devices, and the like. Alternatively, the display control device 10 may be a mobile terminal such as a smartphone or tablet held by a user.

[0013] A virtual object is a virtual content that does not actually exist in a real space. The content is any display information expressed in two or three dimensions. The virtual object may be an object that imitates an object (such as a living thing) that exists in the real world, or an object that imitates an imaginary object that does not exist in the real world. The virtual object may be configured to be operable by a user. As an example, the virtual object may be configured as an object that imitates a soap bubble. For example, when an operation of touching the virtual object (soap bubble) is executed by a user's hand tracking gesture, controller operation, or the like, the virtual object (soap bubble) may be popped, and other objects that have been stored in advance in the virtual object may be expanded and displayed.

[0014] An example of a content image will be described with reference to Fig. 2. The content image IMa shown in Fig. 2(A) is an image in which a virtual object VO resembling a cat is superimposed on a real-life image representing real space. On the other hand, the content image IMb shown in Fig. 2(B) is an image in which a part of the virtual object VO (in this example, the rear part of the cat's body) is expressed to be hidden behind a real object (in this example, a bed) existing in real space.

[0015] The content image IMa is an image in which the virtual object VO is simply superimposed on a real-life image without considering the positional relationship between the virtual object VO and the real object (particularly the positional relationship in the front-back direction (depth direction)). That is, the content image IMa is an image in which the virtual object VO is superimposed on a real-life image without performing occlusion processing (that is, an image expressed in an AR manner). On the other hand, the content image IMb is an image in which the virtual object VO is expressed on the real-life image in a manner with a higher degree of integration by performing occlusion processing while taking into account the positional relationship between the virtual object VO and the real object (that is, an image expressed in an MR manner).

[0016] There are mainly two methods for accurately estimating the positional relationship between a virtual object and a real object in order to generate an image to which an occlusion function is applied, such as the content image IMb: an image recognition type and a spatial recognition type.

[0017] The image recognition type is a method of performing image recognition on image data of real space acquired by a camera or the like (such as an external world information acquisition unit 14 described later) of the display control device 10, and inferring the front-to-back relationship between real objects and virtual objects based on the image recognition results. The image recognition type has the advantage of not needing to acquire real space information (spatial data required for the spatial recognition type described later) in advance. On the other hand, there is a disadvantage in that occlusion display cannot be performed with high accuracy in a real space having a complex structure, since there is a limit to the accuracy of image recognition.

[0018] The spatial recognition type is a method that performs self-location recognition (self-location estimation) and spatial structure recognition (depth estimation) and infers the front-to-back relationship between real objects and virtual objects based on the results of these recognitions.

[0019] Self-location recognition is a technology for recognizing a self-location (i.e., a viewpoint and a line-of-sight direction (orientation) that define a real space displayed on the display unit 12) by matching feature point group data collected in advance in a real space by a sensor (camera, laser, etc.) with information on feature points extracted by a camera or the like (such as an external environment information acquisition unit 14 described later) of the display control device 10. The above-mentioned self-location recognition can be realized by using, for example, an IMU (Inertial Measurement Unit), SLAM (Simultaneous Localization And Mapping), etc.

[0020] The spatial structure recognition is a technology for identifying the position of the real space by applying an algorithm such as SLAM to a photograph capturing the real space displayed on the display unit 12, and comparing the group of feature points obtained with a reference map of the group of feature points extracted and stored in advance from photographs taken at various locations using a similar algorithm, and acquiring spatial structure data corresponding to the position, thereby recognizing the spatial structure in the direction in which the display control device 10 is facing (i.e., the line of sight direction that defines the real space displayed on the display unit 12). Here, the spatial structure data is created in advance by scanning with LiDAR (Light Detection And Ranging), photogrammetry, or the like. Note that, for information within 3 m in the vicinity of the display control device 10, the recognition of the spatial structure may be performed by using, in addition to the spatial structure data, data acquired by sensing the depth with a depth sensor (ToF sensor (LiDAR), etc.) provided in the display control device 10. The spatial structure recognition may be performed by a cloud (i.e., a group of servers connected to be able to communicate with the display control device 10) or locally in the display control device 10.

[0021] According to the spatial recognition type, occlusion display can be performed with higher accuracy than the image recognition type. On the other hand, in order to perform the above-mentioned self-location recognition and spatial structure recognition, spatial data (feature point group data and spatial structure data) regarding the location (area) of the real space displayed on the display unit 12 is required. Examples of the location where the above-mentioned spatial data is prepared (i.e., the location where the spatial data is acquired in advance) include public spaces such as Shibuya Scramble Crossing and semi-public spaces such as inside a shopping mall. However, spatial data is not necessarily acquired in advance for all locations in the real space. In other words, there are locations where the spatial data is not prepared. In addition, when a location where the spatial data is not acquired is displayed on the display unit 12, acquiring (scanning) the spatial data of the location in real time is not realistic in terms of the amount of calculation (calculation time) and the required machine power.

[0022] In consideration of the above, the display control device 10 is configured to estimate the positional relationship between the virtual object and the real object by executing the above-mentioned spatial recognition type processing when the spatial data of the place in the real space displayed on the display unit 12 has been acquired, and to generate a content image (an image expressed in an MR manner) to which the occlusion function is applied, such as the content image IMb shown in (B) of Fig. 2, based on the estimation result. On the other hand, when the spatial data of the place in the real space displayed on the display unit 12 has not been acquired, the display control device 10 is configured to generate a content image (an image expressed in an AR manner) to which the occlusion function is not applied, such as the content image IMa shown in (A) of Fig. 2.

[0023] On the other hand, in such a configuration, when a user (display control device 10) moves from a place where spatial data has not been acquired to a place where spatial data has been acquired, a content image to which the occlusion function is not applied (for example, an image in which a virtual object VO is simply superimposed on a real-life image, such as the content image IMa in FIG. 2A) may suddenly change to a content image to which the occlusion function is applied (for example, an image in which the positional relationship between the real object and the virtual object VO is taken into consideration and the virtual object VO is expressed so that part or all of it is hidden by the real object, such as the content image IMb in FIG. 2B). As a result, the user of the display control device 10 may feel uncomfortable.

[0024] Therefore, the display control device 10 is configured to execute display control for reducing the discomfort described above. In the following, each function of the display control device 10 for implementing the above-mentioned display control will be described in detail.

[0025] As shown in FIG. 1, the display control device 10 includes a storage unit 11, a display unit 12, a location information acquisition unit 13, an outside world information acquisition unit 14, a mode detection unit 15, and a control unit 16.

[0026] The storage unit 11 is a non-transitory storage medium or storage device that stores spatial data including the above-mentioned feature point group data and spatial structure data. The spatial data (feature point group data and spatial structure data) is stored in the storage unit 11, for example, for each location (for example, an area specified by a range of latitude and longitude). The spatial data is downloaded, for example, from a server device or the like managed by a service provider that provides the spatial data, via a communication network (for example, any network such as the Internet), and stored in the storage unit 11.

[0027] The display unit 12 is a display on which a content image is displayed. When the display control device 10 is a wearable device worn on the user's head, the display unit 12 is a display arranged in front of both eyes of the user. The display unit 12 is configured by, for example, a liquid crystal display element, an organic EL (Electro Luminescence) element, or the like. The display unit 12 may have a right-eye display panel arranged in front of the user's right eye, and a left-eye display panel arranged in front of the user's left eye, separately. When the display control device 10 is a mobile terminal such as a smartphone or a tablet, the display unit 12 is configured by a touch panel display.

[0028] The display unit 12 may be configured with an optical see-through type (transmissive type) display, or may be configured with a video see-through type display. In the former case, an optically transmitted image of real space is displayed on the display unit 12. In the latter case, an image (video) acquired (captured) in real time by the external world information acquisition unit 14 (camera) is displayed on the display unit 12.

[0029] The location information acquisition unit 13 acquires location information indicating a location in the real space displayed on the display unit 12. For example, the location information acquisition unit 13 may acquire location information indicating the location (latitude and longitude) of the display control device 10 by using a positioning function such as a GPS provided in the display control device 10. In addition, when the display control device 10 receives wireless radio waves from a wireless access point, the location information acquisition unit 13 may extract an identifier (e.g., SSID of Wi-Fi (registered trademark) or the like) for uniquely identifying the wireless access point from the wireless radio waves, and acquire location information indicating a location corresponding to the extracted identifier (i.e., the location of the wireless access point). In addition, the location information acquisition unit 13 may perform image recognition of the image of the real space acquired by the external world information acquisition unit 14, and when a specific object (e.g., a specific building, sign, bronze statue, monument, etc.) associated with location information in advance is recognized, acquire location information associated with the specific object. The location information acquisition unit 13 can specify a location corresponding to the location information acquired as described above by, for example, referring to table information or the like in which previously prepared location information is associated with a location.

[0030] The external world information acquisition unit 14 acquires an image of the real space. The external world information acquisition unit 14 may be configured by, for example, a camera. The position and orientation of the external world information acquisition unit 14 (camera) are set so as to capture the real space that is visible to the user through the transmissive display (display unit 12). The external world information acquisition unit 14 may also include a depth sensor that measures the distance to each object captured in the image. However, the accuracy of the external world information acquisition unit 14 (depth sensor) included in the display control device 10 is limited, and is lower than the above-mentioned spatial recognition method.

[0031] An image of the real space acquired by the external world information acquisition unit 14 (hereinafter referred to as the "first real image") corresponds to an image of the real space displayed on the display unit 12 (hereinafter referred to as the "second real image"). When the display unit 12 is configured with a video see-through type display, the first real image acquired by the external world information acquisition unit 14 is displayed on the display unit 12, so the first real image and the second real image match. When the display unit 12 is configured with an optical see-through type display, the first real image acquired by the external world information acquisition unit 14 is not displayed on the display unit 12, so the first real image and the second real image do not match completely, but the first real image shows the same content as the second real image. Therefore, the control unit 16 described later can grasp the display content of the display unit 12 (for example, which real object is displayed at which display position on the display unit 12) by referring to the first real image acquired by the external world information acquisition unit 14. The control unit 16 can identify the real object overlapping with the virtual object based on the information thus grasped.

[0032] The mode detection unit 15 continuously detects a mode according to the degree of grasp of the spatial structure of the real space displayed on the display unit 12. In the present embodiment, as an example, the degree of grasp of the spatial structure of the real space includes a first grasp degree and a second grasp degree. That is, in the present embodiment, the mode includes a first mode corresponding to the first grasp degree and a second mode corresponding to the second grasp degree.

[0033] As an example, the first graspability corresponds to a state in which the positional relationship between a real object and a virtual object included in the real space displayed on the display unit 12 cannot be specified with a certain degree of accuracy or higher. On the other hand, the second graspability corresponds to a state in which the positional relationship between a real object and a virtual object included in the real space displayed on the display unit 12 can be specified with a certain degree of accuracy or higher. In this embodiment, the "certain degree of accuracy" is an accuracy obtained by executing the spatial recognition method described above. However, the "certain degree of accuracy" is not necessarily limited to that determined by the above criteria, and can be arbitrarily set by the service provider or the user.

[0034] The first graspability corresponds to a state in which the spatial recognition method cannot be executed. In other words, the first graspability corresponds to a state in which the spatial data corresponding to the location in the real space displayed on the display unit 12 cannot be used (i.e., the spatial data is not stored in the storage unit 11), and corresponds to a relatively low graspability. The first mode according to the first graspability is a mode in which the occlusion function based on the spatial recognition method cannot be applied to the virtual object.

[0035] The second graspability corresponds to a state in which the spatial recognition type technique can be executed. In other words, the second graspability corresponds to a state in which spatial data corresponding to a location in real space displayed on the display unit 12 can be used (i.e., a state in which the spatial data is stored in the storage unit 11), and corresponds to a relatively high graspability. The second mode according to the second graspability is a mode in which an occlusion function based on the above-mentioned spatial recognition type technique can be applied to a virtual object.

[0036] For example, the mode detection unit 15 can detect (identify) the mode as follows. First, the mode detection unit 15 acquires the location information acquired by the location information acquisition unit 13. Next, the mode detection unit 15 determines whether or not spatial data of the location indicated in the location information (i.e., feature point group data and spatial structure data necessary to execute the spatial recognition type technique) is stored in the storage unit 11. That is, the mode detection unit 15 determines whether or not the spatial data of the location indicated in the location information acquired by the location information acquisition unit 13 is available. Then, when the spatial data is unavailable (i.e., when the spatial data of the location is not stored in the storage unit 11), the mode detection unit 15 detects the first mode. On the other hand, when the spatial data is available (i.e., when the spatial data of the location is stored in the storage unit 11), the mode detection unit 15 detects the second mode.

[0037] The control unit 16 controls the display mode of the virtual object on the display unit 12 according to the mode (first mode or second mode in this embodiment) detected by the mode detection unit 15. The control unit 16 determines the display position of the virtual object on the content image displayed on the display unit 12 based on, for example, a predetermined virtual object arrangement rule (for example, the position of the virtual object with respect to the user (i.e., the distance between the user and the virtual object, etc.)), a motion pattern (for example, a pattern of floating in the space in front of the user, etc.), etc. Furthermore, the control unit 16 determines the display mode of the virtual object according to the mode detected by the mode detection unit 15.

[0038] When the mode detected by the mode detection unit 15 changes from the first mode to the second mode, the control unit 16 changes the display mode of the virtual object on the display unit 12 from one display mode corresponding to the first mode to a second display mode corresponding to the second mode via one or more intermediate display modes between the first display mode and the second display mode. A specific example of the processing by the control unit 16 will be described later.

[0039] Next, an example of the operation of the display control device 10 (including a display control method according to an embodiment) will be described with reference to FIG.

[0040] In step S1, the location information acquisition unit 13 acquires location information indicating a location in real space to be displayed on the display unit 12 (that is, a content image).

[0041] In step S2, the mode detection unit 15 detects a mode according to the degree of grasp of the real space displayed on the display unit 12. In this embodiment, the mode detection unit 15 detects (specifies) the mode based on whether or not spatial data of the place indicated in the place information acquired by the place information acquisition unit 13 is stored in the storage unit 11. More specifically, the mode detection unit 15 detects the first mode when the spatial data of the place is not stored in the storage unit 11. On the other hand, the mode detection unit 15 detects the second mode when the spatial data of the place is stored in the storage unit 11.

[0042] In step S3, the control unit 16 determines whether or not the mode detected by the mode detection unit 15 is the first mode. If the detected mode is the first mode (step S3: YES), the control unit 16 executes the process of step S4. On the other hand, if the detected mode is the second mode (step S3: NO), the control unit 16 executes the process of step S5.

[0043] In step S4, the control unit 16 displays the virtual object in a first display manner corresponding to the first mode.

[0044] In this embodiment, the first display mode includes a display mode in which a virtual object is displayed in front of (on the near side of) a real object, regardless of the original positional relationship between the virtual object and the real object. That is, the first display mode includes a display mode in which an occlusion function is not applied, such as the content image IMa shown in (A) of FIG. 2. The first display mode also includes a display mode in which a virtual object is displayed at a size (first size) smaller than the size (second size) in the second display mode according to the second mode. As an example, the second size is a standard size preset for the virtual object, and the first size is a size smaller than the standard size (a size applied when the virtual object is displayed in a reduced size).

[0045] In step S4, first, the control unit 16 determines the position on the content image where the virtual object is to be placed based on a predetermined virtual object placement rule, operation pattern, etc. Subsequently, the control unit 16 displays the virtual object in the first display mode. The virtual object VO in the content image IM0 of FIG. 4 shows an example of a virtual object displayed in the first display mode. In this example, a part of the virtual object VO overlaps with the real object RO included in the content image IM0. Here, it is assumed that the position of the virtual object VO in the depth direction determined based on the above-described placement rule and operation pattern, etc. is actually located behind (deeper side) the real object RO. However, in the first mode, spatial data cannot be used, and occlusion processing cannot be executed with high precision. Therefore, the control unit 16 displays the virtual object VO in the first display mode. That is, the control unit 16 displays the virtual object VO with a volume "kV (0 < k < 1)" (first size) smaller than the originally set original volume "V". Further, regardless of the original positional relationship between the virtual object VO and the real object RO, the control unit 16 displays the virtual object VO in front of (nearer side) the real object RO.

[0046] By executing the process of step S4, the content image IM0 including the virtual object VO displayed in the first display mode together with the real space is displayed on the display unit 12. The user can experience AR (augmented reality) by visually recognizing the content image IM0 displayed on the display unit 12.

[0047] After the process of step S4 is executed, the processes of steps S1 to S3 described above are executed continuously (for example, periodically at predetermined intervals). For example, when the user moves with the display control device 10 and enters a place where corresponding spatial data exists (for example, the above-mentioned public space, semi-public space, etc.), the second mode is detected by the mode detection unit 15 in step S2. Alternatively, even if the user does not move, when a spatial scan of a place in real space displayed on the display unit 12 is completed and spatial data of the place is acquired, the second mode may be detected by the mode detection unit 15 in step S2 executed thereafter. In such a case, the determination result of step S3 is "NO", and therefore the process of step S5 is executed.

[0048] In step S5, the control unit 16 does not immediately display the virtual object VO in the second display mode corresponding to the second mode, but changes the display mode of the virtual object VO stepwise (continuously) from the first display mode corresponding to the first mode to the second display mode corresponding to the second mode via one or more intermediate display modes.

[0049] In this embodiment, the second display mode includes a display mode in which a part of the virtual object VO hidden by the real object RO is displayed transparently. That is, the second display mode includes a display mode in which an occlusion function is applied, as in the content image IMb shown in (B) of FIG. 2. The control unit 16 can display the virtual object VO in a display mode in which the occlusion function is applied as described above, based on the information acquired by the external world information acquisition unit 14 (in this embodiment, image data of the real space acquired by the camera and depth information extracted by the depth sensor) and the spatial data (feature point group data and spatial structure data) stored in the storage unit 11. That is, the control unit 16 can specify the positional relationship between the real object RO and the virtual object VO by using the spatial data, and perform an occlusion process (i.e., a process of transparently displaying (hiding) the part of the virtual object VO hidden by the real object RO) based on the specified positional relationship. In addition, the second display mode includes a display mode in which the virtual object VO is displayed in a size (second size) larger than the size (first size) in the first display mode.

[0050] A specific example of the process of step S5 will be described with reference to FIG. 4. As an example, the control unit 16 gradually changes the display mode of the virtual object VO from the first display mode to the second display mode so that the display mode of the virtual object VO becomes the second display mode at a time (t=T) when a transition period T determined in advance has elapsed, starting from a time (t=0) when the mode detected by the mode detection unit 15 has changed from the first mode to the second mode. For example, if the mode first detected immediately after starting the display control device 10 is the second mode, the control unit 16 may execute the following process starting from a time (t=0) when the second mode is first detected. In the following description, the size applied to the virtual object VO displayed on the content image at a certain time t is represented as "V(t)", and the transparency applied to the part of the virtual object VO hidden by the real object RO at a certain time t is represented as "alpha(t)". ​​A transparency of "0" indicates a completely opaque state, and a transparency of "100" indicates a completely transparent state.

[0051] At the starting point (t = 0), in the same manner as the process of step S4, the control unit 16 displays the virtual object VO in the first display mode. That is, the control unit 16 sets the size (volume) of the virtual object VO to "V(0) = kV". Further, the control unit 16 sets the transparency of the portion of the virtual object VO hidden by the real object RO (i.e., the portion of the virtual object VO overlapping with the real object RO) to "alpha(0) = 0". As a result, the content image IM0 shown in FIG. 4 is displayed on the display unit 12.

[0052] Subsequently, at the time point (t = t1) when a predetermined period t1 (t1 < T) has elapsed from the starting point, the control unit 16 displays the virtual object VO in an intermediate display mode (the first intermediate display mode) between the first display mode and the second display mode. As an example, the control unit 16 sets the size (volume) of the virtual object VO to a value "V(t1)" that is larger than "kV" and smaller than "V". That is, "kV < V(t1) < V" holds. Further, the control unit 16 sets the transparency of the portion of the virtual object VO hidden by the real object RO (i.e., the portion of the virtual object VO overlapping with the real object RO) to a value "alpha(t1)" that is larger than "0" and smaller than "100". That is, the control unit 16 displays the portion of the virtual object VO hidden by the real object RO semi-transparently so that "0 < alpha(t1) < 100" holds. As a result, the content image IM1 shown in FIG. 4 is displayed on the display unit 12.

[0053] Subsequently, at the time point (t = t2) when a predetermined period t2 (t1 < t2 < T) has elapsed from the starting point, the control unit 16 displays the virtual object VO in an intermediate display mode (second intermediate display mode) between the first display mode and the second display mode. As an example, the control unit 16 sets the size (volume) of the virtual object VO to a value "V(t2)" that is larger than "V(t1)" and smaller than "V". That is, "V(t1) < V(t2) < V" holds. Also, the control unit 16 sets the transparency of the portion of the virtual object VO hidden by the real object RO (i.e., the portion of the virtual object VO overlapping with the real object RO) to a value "alpha(t2)" that is larger than "alpha(t1)" and smaller than "100". That is, the control unit 16 displays the portion of the virtual object VO hidden by the real object RO semi-transparently so that "alpha(t1) < alpha(t2) < 100" holds. As a result, the content image IM2 shown in FIG. 4 is displayed on the display unit 12.

[0054] Subsequently, at the time point (t = T) when the transition period T has elapsed from the starting point, the control unit 16 displays the virtual object VO in the second display mode. That is, the control unit 16 sets the size (volume) of the virtual object VO to "V" (second size). Also, the control unit 16 sets the transparency of the portion of the virtual object VO hidden by the real object RO (i.e., the portion of the virtual object VO overlapping with the real object RO) to "100". That is, the control unit 16 makes the portion of the virtual object VO hidden by the real object RO completely invisible. As a result, the content image IM3 shown in FIG. 4 is displayed on the display unit 12.

[0055] In the above example, two intermediate display modes (t = t1, t2) are exemplified, but there may be one intermediate display mode. Alternatively, three or more intermediate display modes may be set in order to change the virtual object VO more smoothly from the first display mode to the second display mode.

[0056] In the display control device 10 described above, the mode detection unit 15 detects a mode corresponding to the degree of understanding of the spatial structure of the real space displayed on the display unit 12. Then, the control unit 16 displays the virtual object VO on the display unit 12 in a display mode corresponding to the mode. More specifically, when the first mode is changed to the second mode, the control unit 16 changes the display mode of the virtual object VO from a first display mode corresponding to the first mode (for example, the content image IM0 in FIG. 4) via one or more intermediate display modes (for example, the content images IM1 and IM2 in FIG. 4) to a second display mode corresponding to the second mode (for example, the content image IM3 in FIG. 4). That is, when the mode is changed, the control unit 16 does not immediately reflect the display mode (second display mode) corresponding to the latest detected mode (second mode) on the virtual object VO, but changes it stepwise from the display mode (first display mode) corresponding to the mode before the change (first mode) to the display mode (second display mode) corresponding to the mode after the change (second mode). The above process makes it possible to prevent the display state of the virtual object VO from changing suddenly when the mode is changed, thereby reducing the sense of discomfort felt by the user.

[0057] Furthermore, the control unit 16 may gradually change the display mode of the virtual object VO from the first display mode to the second display mode such that the display mode of the virtual object VO becomes the second display mode at a time point when a predetermined transition period T has elapsed, starting from a time point t0 when the mode detected by the mode detection unit 15 changes from the first mode to the second mode. With this configuration, the display mode of the virtual object VO can be changed gradually over the predetermined transition period T, and therefore the display mode of the virtual object VO can be changed without giving a sense of incongruity to the user.

[0058] Furthermore, the first display mode may include a display mode in which the virtual object VO is displayed in front of the real object RO regardless of the original positional relationship between the virtual object VO and the real object RO (i.e., a display mode in which the occlusion function is not applied), the second display mode may include a display mode in which a portion of the virtual object VO hidden by the real object RO is displayed transparently (i.e., a display mode in which the occlusion function is applied), and the one or more intermediate display modes may include a display mode in which a portion of the virtual object VO hidden by the real object RO is displayed semi-transparently (e.g., the content images IM1 and IM2 in FIG. 4). According to this configuration, the portion of the virtual object VO hidden by the real object RO is not immediately displayed transparently (not displayed), but the transparency is gradually increased, thereby making it possible to change the display mode of the virtual object VO without giving a sense of incongruity to the user.

[0059] Furthermore, the mode detection unit 15 may determine whether or not spatial data related to the spatial structure of the place indicated in the place information acquired by the place information acquisition unit 13 is available, detect the first mode when the spatial data is unavailable, and detect the second mode when the spatial data is available. Furthermore, when displaying the virtual object VO in the second display mode, the control unit 16 may specify a part of the virtual object VO that is hidden by the real object RO based on the spatial data. With this configuration, it is possible to easily and reliably execute the mode detection process based on the availability of spatial data for specifying the part of the virtual object VO that is hidden by the real object RO.

[0060] Moreover, the first display mode may include a display mode in which the virtual object VO is displayed in a first size, the second display mode may include a display mode in which the virtual object VO is displayed in a second size different from the first size, and the one or more intermediate display modes may include a display mode in which the virtual object VO is displayed in a size between the first size and the second size. According to this configuration, when the mode is changed, the size of the virtual object VO is changed stepwise instead of abruptly, thereby reducing the sense of discomfort felt by the user. Furthermore, the second size may be larger than the first size. In this case, in the first display mode in which the occlusion function is not applied, the display size of the virtual object VO is relatively small, thereby reducing the sense of discomfort felt by the user. That is, by preventing a part of the virtual object VO that should be hidden by the real object RO (i.e., that should be located behind the real object RO) from being displayed larger in front of the real object RO, the sense of discomfort felt by the user can be effectively reduced.

[0061] (Modification) Note that variations of the first display mode and the second display mode are not limited to the above embodiment. For example, the first display mode may include a display mode in which the virtual object VO is displayed in a first color tone (for example, monochrome display, etc.), the second display mode may include a display mode in which the virtual object VO is displayed in a second color tone (for example, color display, etc.) different from the first color tone, and one or more intermediate display modes may include a display mode in which the virtual object VO is displayed in a third color tone between the first color tone and the second color tone. In this case, when the mode is changed, the color tone of the virtual object VO is changed stepwise rather than abruptly, thereby reducing the sense of incongruity felt by the user. Also, as an example, the first color tone may be expressed by a color tone that is less noticeable than the second color tone. In this case, in a state in which the virtual object VO cannot be displayed by applying the occlusion function (i.e., the first display mode), the sense of incongruity felt by the user can be effectively reduced by making the virtual object VO less noticeable.

[0062] Moreover, the first display mode may include a display mode in which the virtual object VO is displayed in a first shape, the second display mode may include a display mode in which the virtual object VO is displayed in a second shape different from the first shape, and the one or more intermediate display modes may include a display mode in which the virtual object VO is displayed in a third shape between the first shape and the second shape. For example, the first shape is a shape different from the original shape of the virtual object VO (for example, a shape deformed into a simple shape such as a sphere), and the second shape is the original shape of the virtual object VO (i.e., a shape previously set for the virtual object VO). In this case, when the mode is changed, the shape of the virtual object VO is changed stepwise rather than abruptly, thereby reducing the sense of incongruity felt by the user. Furthermore, as in the above example, in a state in which the virtual object VO cannot be displayed by applying the occlusion function (i.e., the first display mode), the virtual object VO is displayed in a shape different from the original shape (the first shape), thereby allowing the user to easily understand that the occlusion function is not applied to the virtual object VO. As a result, it is possible to effectively suppress the sense of incongruity felt by the user when the virtual object VO is displayed in front of the real object RO.

[0063] In the above embodiment, the relatively low level of comprehension of the spatial structure is defined as the first comprehension degree, and the relatively high level of comprehension of the spatial structure is defined as the second comprehension degree. However, the relatively high level of comprehension of the spatial structure may be defined as the first comprehension degree, and the relatively low level of comprehension of the spatial structure may be defined as the second comprehension degree. That is, in the above embodiment, when changing from the second mode in which the comprehension of the spatial structure is relatively high to the first mode in which the comprehension of the spatial structure is relatively low, the display mode of the virtual object may be changed from the second display mode to the first display mode via one or more intermediate display modes. That is, in the example of FIG. 4, when the mode detected by the mode detection unit 15 changes from the second mode to the first mode, the control unit 16 may change the display mode of the virtual object VO from the second display mode (content image IM3) to the first display mode (content image IM0) via intermediate display modes (content images IM2, IM1). In this case, too, it is possible to prevent the display mode of the virtual object from changing suddenly, thereby reducing the sense of incongruity felt by the user.

[0064] Furthermore, the control unit 16 may smoothly change the display mode of the virtual object VO when the mode changes according to the degree of grasp of the spatial structure, regardless of the mode transition direction. For example, in the above embodiment, when the first mode is changed to the second mode, the control unit 16 may change the display mode of the virtual object VO from the first display mode to the second display mode via one or more intermediate display modes, and when the second mode is changed to the first mode, the control unit 16 may change the display mode of the virtual object VO from the second display mode to the first display mode via one or more intermediate display modes. In this case, the display mode of the virtual object VO can be smoothly transitioned between modes according to the degree of grasp of the spatial structure, and the sense of discomfort felt by the user can be effectively reduced.

[0065] In addition, in the above embodiment, the modes according to the degree of comprehension of the spatial structure are only two, the first mode and the second mode, but three or more modes may be used. In this case, when the mode detected by the mode detection unit 15 changes from one mode (corresponding to the first mode in the above embodiment) to another mode (corresponding to the second mode in the above embodiment), the control unit 16 may change the display mode of the virtual object VO from the display mode corresponding to the one mode (corresponding to the first display mode in the above embodiment) to the display mode corresponding to the other mode (corresponding to the second display mode in the above embodiment) via one or more intermediate display modes.

[0066] In addition, the control unit 16 may use the above-mentioned spatial recognition type (self-position recognition and spatial structure recognition) technique to perform physical expression other than occlusion processing. Some concrete examples of such physical expression are shown below. -Roll a ball, a virtual object, on the floor in real space. - Making a robot, which is a virtual object, walk on a floor in real space. - Bounce a virtual ball off a wall in real space. Colliding virtual objects with real objects (for example, deforming or destroying virtual objects on collision). -Move virtual objects to avoid real objects.

[0067] In the above embodiment, a user terminal such as a wearable terminal or a mobile terminal (e.g., a smartphone, a tablet, etc.) used by a user functions as the display control device 10. However, some or all of the functional elements other than the display unit 12 of the display control device 10 shown in FIG. 1 may be implemented in a device (e.g., a server configured to be able to communicate with the user terminal) other than the user terminal having the display unit 12. When the external world information acquisition unit 14 is implemented in a server other than the user terminal, the external world information acquisition unit 14 may acquire information acquired by a camera, a depth sensor, etc. provided in the user terminal by receiving the information from the user terminal. For example, when all of the functional elements other than the display unit 12 of the above-mentioned display control device 10 are implemented in a server, the server can function as a display control device that performs display control of the display unit 12 (i.e., the display unit provided in the user terminal) on which virtual objects are displayed together with the real space. Also, for example, when some of the functional elements other than the display unit 12 of the above-mentioned display control device 10 are implemented in a server, a system including the user terminal and the server can function as a display control device that performs display control of the display unit 12 on which virtual objects are displayed together with the real space.

[0068] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

[0069] Functions include, but are not limited to, judging, determining, assessing, calculating, processing, deriving, investigating, searching, verifying, receiving, sending, outputting, accessing, resolving, selecting, choosing, establishing, comparing, anticipating, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.

[0070] For example, the display control device 10 according to an embodiment of the present disclosure may function as a computer that performs the display control method of the present disclosure. Fig. 5 is a diagram showing an example of a hardware configuration common to the display control device 10 according to an embodiment of the present disclosure. Each of the display control devices 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0071] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the display control device 10 may be configured to include one or more of the devices shown in FIG. 5, or may be configured to exclude some of the devices.

[0072] Each function in the display control device 10 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communication via a communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and the storage 1003.

[0073] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like.

[0074] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the read out programs. As the program, a program that causes a computer to execute at least a part of the operations described in the above-mentioned embodiment is used. For example, each functional unit (e.g., the control unit 16, etc.) of the display control device 10 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from a network via a telecommunication line.

[0075] The memory 1002 is a computer-readable recording medium, and may be configured by at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software module, etc. that can be executed to implement a display control method according to an embodiment of the present disclosure.

[0076] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0077] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0078] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0079] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0080] Furthermore, the display control device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0081] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as a modified or altered form without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

[0082] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0083] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0084] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0085] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0086] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0087] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0088] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0089] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0090] The names used for the above-mentioned parameters are not limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not limiting in any way.

[0091] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0093] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0094] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0095] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]

[0096] 10...display control device, 11...memory unit, 12...display unit, 13...location information acquisition unit, 14...external world information acquisition unit, 15...mode detection unit, 16...control unit, IMa, IMb, IM0, IM1, IM2, IM3...content image, RO...real object, VO...virtual object.

Claims

1. A display control device that controls display of a display unit on which a virtual object is displayed together with a real space, a mode detection unit that detects a mode according to a degree of comprehension of a spatial structure of the real space displayed on the display unit; a control unit that controls a display manner of the virtual object on the display unit in accordance with the mode detected by the mode detection unit; Equipped with When the mode detected by the mode detection unit is changed from a first mode corresponding to a first graspability degree to a second mode corresponding to a second graspability degree, the control unit changes a display state of the virtual object on the display unit from a first display state corresponding to the first mode to a second display state corresponding to the second mode, via one or more intermediate display states between the first display state and the second display state. Display control device.

2. the control unit gradually changes the display state of the virtual object from the first display state to the second display state such that the display state of the virtual object becomes the second display state at a time point when a predetermined transition period has elapsed from a time point when the mode detected by the mode detection unit changes from the first mode to the second mode. The display control device according to claim 1 .

3. the first grasp degree is a grasp degree corresponding to a state in which a positional relationship between a real object included in the real space displayed on the display unit and the virtual object cannot be specified with a certain accuracy or higher, the second graspability level is a graspability level corresponding to a state in which a positional relationship between a real object and the virtual object included in the real space displayed on the display unit can be specified with the certain accuracy or higher, the first display manner includes a display manner in which the virtual object is displayed in front of the real object regardless of an original positional relationship between the virtual object and the real object, the second display manner includes a display manner in which a portion of the virtual object that is hidden by the real object is displayed transparently; the one or more intermediate display modes include a display mode in which a portion of the virtual object that is hidden by the real object is displayed semi-transparently; The display control device according to claim 1 .

4. A location information acquisition unit that acquires location information indicating a location in the real space to be displayed on the display unit, The mode detection unit is determining whether spatial data relating to a spatial structure of the place indicated in the place information acquired by the place information acquisition unit is available; Detecting the first mode when the spatial data is unavailable; Detecting the second mode when the spatial data is available; the control unit, when displaying the virtual object in the second display manner, specifies a portion of the virtual object that is hidden by the real object based on the spatial data. The display control device according to claim 3 .

5. the first display manner includes a display manner in which the virtual object is displayed in a first size, the second display manner includes a display manner in which the virtual object is displayed at a second size different from the first size, the one or more intermediate display modes include a display mode in which the virtual object is displayed at a size between the first size and the second size; The display control device according to claim 1 .

6. The second size is greater than the first size. The display control device according to claim 5 .

7. the first display manner includes a display manner in which the virtual object is displayed in a first color tone, the second display mode includes a display mode in which the virtual object is displayed in a second color tone different from the first color tone, the one or more intermediate display modes include a display mode in which the virtual object is displayed in a third color tone between the first color tone and the second color tone; The display control device according to claim 1 .

8. the first display mode includes a display mode in which the virtual object is displayed in a first shape, the second display mode includes a display mode in which the virtual object is displayed in a second shape different from the first shape, the one or more intermediate display modes include a display mode in which the virtual object is displayed in a third shape between the first shape and the second shape; The display control device according to claim 1 .

Images