Contents providing device

Abstract

A server 10 according to one embodiment comprises: a detecting unit 11 for detecting a real object, which is an object included in a real space image imaged by an HMD 20 used by a first user; a registering unit 12 for registering the real object detected by the detecting unit 11, in association with the first user; an associating unit 14 for associating virtual content that does not exist in a real space to the real object registered by the registering unit 12; and a display control unit 15 for causing the content to be displayed together with the real space on a display unit 22 of the HMD 20 used by the first user. If the real object is contained in the real space displayed by the display unit 22, the display control unit 15 causes the content to be displayed in association with the real object.

Description

[Technical field]

[0001] One aspect of the present invention relates to a content providing device. [Background technology]

[0002] Patent Literature 1 discloses a mechanism for providing a user with augmented reality (AR) or mixed reality (MR) by superimposing a virtual object on an image (video) showing a real space displayed on a display device. Patent Literature 1 discloses a display device that, when a marker image registered in advance on the system side is recognized in a captured image, displays a virtual object associated with the marker image in a display position associated with the marker image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-203382 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a mechanism for providing users with the above-mentioned augmented reality or mixed reality, there is a demand for a mechanism that can improve the convenience of the user's AR or MR experience.

[0005] Therefore, an object of one aspect of the present invention is to provide a content providing device that can improve the convenience of a user's AR or MR experience. [Means for solving the problem]

[0006] A content providing device according to one aspect of the present invention includes a detection unit that detects a real object, which is an object included in a real space image captured by a user terminal used by a first user, a registration unit that registers the real object detected by the detection unit in association with the first user, an association unit that associates virtual content that does not exist in the real space with the real object registered by the registration unit, and a display control unit that displays the content together with the real space on a display unit of the user terminal used by the first user, and when a real object is included in the real space displayed on the display unit, the display control unit displays the content in association with the real object.

[0007] In a content providing device according to one aspect of the present invention, an object included in a real space image captured by a user terminal of a first user is automatically registered as a real object associated with the first user. That is, an object present around the user when the first user uses the user terminal (in other words, an object that catches the first user's eye) can be automatically registered as a real object. Therefore, the first user does not need to manually register an object present around the user, and the first user's effort is reduced. Then, when a real object is displayed on the display unit of the user terminal, virtual content is displayed in association with the real object. According to the above mechanism, it is possible to display virtual content in association with an object that is likely to be seen by the first user when using the user terminal, and therefore content can be effectively presented to the first user. As a result, the convenience of the first user in the AR experience or MR experience can be improved. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to provide a content providing device that can improve the convenience of a user's AR or MR experience. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating an example of a functional configuration of a content providing system according to an embodiment. [Diagram 2] FIG. 10 is a diagram illustrating an example of list information. [Diagram 3] FIG. 13 is a diagram showing an example of a content display. [Figure 4] FIG. 2 is a sequence diagram showing an example of an operation of the content providing system. [Diagram 5] 5 is a flowchart showing an example of the process of step S5 in FIG. 4. [Figure 6] FIG. 2 illustrates an example of a hardware configuration of a server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or corresponding elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0011] 1 is a diagram showing an example of a content providing system 1 according to an embodiment. The content providing system 1 is a computer system that provides virtual content (virtual objects) to be displayed on a display unit of a user terminal used by a user. In other words, the content providing system 1 is a system that provides an AR experience or an MR experience to a user by displaying virtual content together with a real space on the display unit of the user terminal.

[0012] As an example, the content providing system 1 includes a server 10 (content providing device), an HMD (Head Mounted Display) 20 which is a user terminal used by a first user, a wrist DB 30, and an operation terminal 40 which is used by a second user different from the first user. The server 10 is communicably connected to the HMD 20, the wrist DB 30, and the operation terminal 40 via any wired or wireless communication network.

[0013] An overview of the processing of the content providing system 1 will be described. The HMD 20 worn on the head of the first user has an imaging function (recognition unit 21) that images the real space in front of the HMD 20 to obtain a real space image, and detects (recognizes) an object included in the real space image. The server 10 registers a real object, which is an object detected from the real space image, in association with the first user. For example, the server 10 stores information on the real object in the list DB 30. Then, when the second user operates the operation terminal 40 to request list information indicating a list of real objects associated with the first user from the server 10, the server 10 transmits the list information to the operation terminal 40, and the list of real objects is displayed on the operation terminal 40. The second user operates the operation terminal 40 to select one real object (hereinafter referred to as "real object X") and specify virtual content (hereinafter referred to as "content Y") to be associated with the real object. The information inputted in the operation terminal 40 (i.e., information indicating the real object X and the content Y) is transmitted to the server 10. The server 10 controls the display of the content in the HMD 20, thereby displaying the content together with the real space on the display unit 22 of the HMD 20. Specifically, when a real object X is included in the real space displayed on the display unit 22 of the HMD 20, the server 10 displays the content Y in association with the real object X.

[0014] In FIG. 1, only one HMD 20 is illustrated as a target for which the server 10 performs content display control, but the server 10 may be configured to perform display control for multiple HMDs 20. In this case, the server 10 executes the above-mentioned real object registration process (i.e., a process of associating a real object with a first user and storing it in the list DB 30), whereby list information for each of multiple users (first users) is stored in the list DB 30. When list information for multiple users is stored in the list DB 30 in this manner, a second user using the operation terminal 40 may select a first user to whom content is to be transmitted and request list information for the selected first user from the server 10. The server 10 may be configured by a single computer device, or may be configured by multiple computer devices that can communicate with each other.

[0015] The HMD 20 is a device worn on the user's head. The form of the HMD 20 is not limited to a specific form. The HMD 20 may take various forms, such as goggles, glasses (eyeglasses), and hats. The HMD 20 is, for example, smart glasses. In this embodiment, the HMD 20 is AR glasses having a function of providing an augmented reality (AR) to the user. For example, the HMD 20 is a see-through type of glasses configured to allow the user to view a real space (outside world) and virtual content superimposed on the real space. Note that the HMD 20 is not limited to the above, and may be an MR device such as MR glasses having a function of providing a mixed reality (MR) to the user.

[0016] The HMD 20 includes a recognition unit 21 and a display unit 22 .

[0017] The recognition unit 21 has a function of capturing an image of the real space in front of the HMD 20 and acquiring the real space image. The real space image may be a still image or a moving image (video). The recognition unit 21 also has a function of detecting (object recognition) an object included in the real space image (i.e., an object captured as a subject in the real space image). The recognition unit 21 is composed of, for example, a camera and a program for analyzing the real space image captured by the camera. The recognition unit 21 detects an object included in the real space image by, for example, performing at least one of image recognition (2D image recognition), photogrammetry, and 3D scanning on the real space image.

[0018] In this embodiment, the recognition unit 21 acquires (generates) 3D data (3D model) of the detected object. The 3D data of the object detected by the recognition unit 21 is transmitted to the server 10 together with time information indicating the detection time and location information indicating the detection location. The time information is information indicating the time (e.g., date and time) when the real space image in which the object is detected was captured. The location information is information indicating the real space location indicated by the real space image in which the object is detected. The location information may include position information indicating latitude, longitude, altitude, etc. acquired by a GPS sensor or the like provided in the HMD 20. Alternatively, for example, when the HMD 20 has a function of wirelessly communicating with a wireless LAN access point installed in a private space such as the home of the first user or a public space such as a store or public facility, an ID (e.g., BSSID, etc.) for identifying the wireless LAN access point connected to the HMD 20 for communication may be used as the location information. In this case, the installation location of the wireless LAN access point identified by the ID is used as the detection location of the object.

[0019] The display unit 22 is a display arranged in front of both eyes of the user. The display unit 22 is composed of, for example, a liquid crystal display element, an organic EL (Electro Luminescence) element, or the like. The display unit 22 may have a display panel for the right eye arranged in front of the right eye of the user and a display panel for the left eye arranged in front of the left eye of the user separately. The display unit 22 may be composed of an optical see-through (transmissive) display, or may be composed of a video see-through display. In the former case, a transmitted image of a real space optically transmitted (i.e., a real space in front of the display unit 22) is displayed on the display unit 22. In the latter case, a video image (image) acquired (captured) in real time by a camera (recognition unit 21) provided in the HMD 20 is displayed on the display unit 22. In addition, the display unit 22 displays virtual content (i.e., content distributed from the server 10) superimposed on an image (transmissive image or video image) of the real space.

[0020] The content is display information that indicates any information. The form of the content is not limited to a specific form, but examples of the content include images (still images or moving images), text information, graphic information, 3D models, etc.

[0021] The server 10 includes a detection unit 11, a registration unit 12, a list presentation unit 13, an association unit 14, and a display control unit 15.

[0022] The detection unit 11 detects a real object that is an object included in a real space image captured by the HMD 20. The real object is not limited to a specific object, but examples of the real object include belongings of the first user, such as a PC monitor (display), a pen, a cup, a smartphone, and a poster attached to the wall of the first user's room. Here, the "detection of a real object" performed by the detection unit 11 means grasping (acquiring) characteristics of the real object, such as a shape, color, pattern, etc., so that the real object can be detected when the real object exists in the real space displayed on the display unit 22 of the HMD 20 later. In the present embodiment, as an example, the detection unit 11 acquires 3D data of the real object as the characteristics of the real object. In addition, the detection unit 11 may detect the real object (recognize the object) by using not only the 3D data acquired by the HMD 20 but also a 2D image acquired by the RGB camera of the HMD 20.

[0023] In this embodiment, the detection unit 11 detects a real object by acquiring 3D data, which is a result of detection (object recognition) by the recognition unit 21 of the HMD 20, from the HMD 20. In this embodiment, the real space image is an image obtained by capturing an image of the real space in front of the HMD 20 worn on the head of the first user. Therefore, in this embodiment, the detection unit 11 detects an object included in the real space image corresponding to the field of view of the first user as a real object. In other words, an object seen by the first user while using (wearing) the HMD 20 is detected as a real object.

[0024] The method of detecting a real object by the detection unit 11 is not limited to the above. For example, the detection unit 11 may obtain a real space image captured by the HMD 20 from the HMD 20, and detect a real object based on a result of performing at least one of image recognition, photogrammetry, and 3D scanning on the real space image. That is, the detection of a real object may be performed by the HMD 20 or by the server 10. Furthermore, the detection of a real object may be shared between the HMD 20 and the server 10.

[0025] The registration unit 12 registers the real object detected by the detection unit 11 in association with the first user. That is, the registration unit 12 registers the first user, who is the user (owner) of the HMD 20 that captured the real space image in which the real object was detected, in the list DB 30 in association with the real object. By executing the above process for each of the multiple real objects, information (list information) of the multiple real objects associated with the first user is automatically stored in the list DB 30. Some processing examples of the registration unit 12 will be described below.

[0026] (First example of registration process) The real space image captured by the HMD 20 (recognition unit 21) may include only a part of an object. Furthermore, a real space image from a certain angle (direction) may only allow analysis of a part of the whole object that is visible when viewed from the certain angle (direction). That is, the 3D data acquired by the detection unit 11 (in this embodiment, the 3D data acquired by the recognition unit 21) may be incomplete data corresponding to only a part of the object. Therefore, the registration unit 12 may acquire and register complete 3D data corresponding to the whole object by performing the following process.

[0027] That is, the registration unit 12 may identify an item (for example, a product uniquely identified by the model number of the PC monitor) corresponding to an object (real object) obtained from a real space image by comparing 3D data corresponding to a part of the object (for example, a part of a PC monitor (display)) with 3D data of multiple items prepared in advance. Here, the former 3D data is incomplete 3D data acquired by the recognition unit 21. Meanwhile, the latter 3D data is complete 3D data corresponding to the whole of the item stored in a database that can be referenced from the server 10 (for example, a database that stores product information such as model numbers, product names, and 3D data of multiple items). Then, the registration unit 12 may register the 3D data of the identified item as information of the real object. According to such a process, even if the 3D data of the real object obtained based on the real space image acquired by the HMD 20 is incomplete, the complete 3D data can be acquired and registered by referring to the product information prepared in advance.

[0028] The 3D data registered in this manner is used to detect a real object included in the real space displayed on the display unit 22 when performing a display control (i.e., a control for displaying content on the display unit 22 of the HMD 20 in association with the real object) described below. Therefore, according to the first example of the registration process, by registering the complete 3D data as information on the real object, it becomes possible to perform the above-mentioned display control with high accuracy. That is, it becomes possible to detect a real object included in the real space displayed on the display unit 22 with high accuracy based on the complete 3D data. As a result, it is possible to prevent the occurrence of a problem in which a real object is not detected despite being displayed on the display unit 22, and content is not displayed in association with the real object.

[0029] (Second example of registration process) The registration unit 12 may register, as part of the information of the real object, unique information indicating characteristics unique to the object together with item information indicating the item corresponding to the object detected from the real space image. The item information is information indicating characteristics common to the item (product) corresponding to the detected object. The item information may include, for example, information such as the product name, model number, and shape (3D data). On the other hand, the unique information is information indicating characteristics unique to the object actually detected. Examples of the unique information include information indicating the shape and arrangement of scratches, stains, stickers, sticky notes, characters, etc. on the object. Such unique information may be extracted by performing image analysis or the like on the real space image by the detection unit 11. For example, the detection unit 11 may identify the item (product) corresponding to the detected object, and then extract a difference by comparing 3D data registered in advance for the item with the object actually captured in the real space image, and acquire the extracted difference as unique information. Then, the registration unit 12 may register the unique information acquired by the detection unit 11 as part of the information of the real object.

[0030] The unique information registered in this manner is used to detect a real object included in the real space displayed on the display unit 22 when performing a display control (i.e., a control for displaying content on the display unit 22 of the HMD 20 in association with the real object) described below. Therefore, according to the second example of the registration process, when determining whether or not a real object exists in the real space displayed on the display unit 22, the unique information described above is used, making it possible to detect a real object included in the real space displayed on the display unit 22 with even greater accuracy.

[0031] (Third example of registration process) An object that is detected only once within a predetermined period (e.g., one day) may be an object that the first user just happens to see while using the HMD 20 (i.e., an object that the first user rarely sees in everyday life). Even if such an object is registered as a real object associated with the first user and virtual content is associated with the real object, the first user is very unlikely to see the real object, so there is a problem that the content cannot be presented appropriately to the first user. Therefore, in order to present content appropriately to the first user, it is preferable to register an object that the first user sees with a certain frequency or more during the first user's everyday life (when using the HMD 20) as a real object associated with the first user.

[0032] Therefore, the registration unit 12 may register an object whose frequency of detection from real space images captured within a predetermined period is equal to or higher than a predetermined threshold as a real object associated with the first user. For example, if the recognition unit 21 detects an object included in the real space image and the object is set not to be detected for a certain period (e.g., one minute) even if the object is included in the real space image, the number of detections in the predetermined period (e.g., the number of detections per hour) may be used as the detection frequency. If the object continues to be detected while the object is included in the real space image as a subject, the period during which the object was detected within the predetermined period (e.g., the detection period per hour) may be used as the detection frequency. In this embodiment, the registration unit 12 can calculate the detection frequency by referring to the time information transmitted from the HMD 20 to the server 10 as described above.

[0033] According to the third example of the registration process, it is possible to register, as a real object associated with the first user, only an object that is more than a certain probability of being seen by the first user while the first user is using the HMD 20. Therefore, when a setting is made to associate content with a real object, it is possible to present the content to the first user at an appropriate timing (frequency).

[0034] (Fourth example of registration process) In order to more reliably present content to the first user, it is preferable to register the first user's belongings that are likely to be viewed by the first user on a regular basis as real objects associated with the first user. Here, an object detected while the first user is using the HMD 20 in a private space such as the first user's home or room is likely to be the first user's belongings. On the other hand, an object detected while the first user is using the HMD 20 in a place other than the above-mentioned private space (for example, a place that the first user passes by on the way to work or school, a public space such as a workplace or school, etc.) is likely not the first user's belongings.

[0035] Therefore, the registration unit 12 may acquire location information indicating the location where the object was detected (i.e., the location where the real space image where the object was detected was captured by the HMD 20), and determine whether or not to register the object detected from the real space image as a real object associated with the first user based on the location information. In this embodiment, as described above, the location information is transmitted from the HMD 20 to the server 10.

[0036] For example, when location information of an object (e.g., a location indicated by position coordinates acquired by GPS) indicates a location previously registered as a private space of the first user, the registration unit 12 may register the object as a real object associated with the first user. On the other hand, when location information of an object indicates a location not registered as a private space of the first user, the registration unit 12 may not register the object as a real object associated with the first user.

[0037] Furthermore, among the belongings of the first user, an object carried by the first user, such as a smartphone, is considered to be detected in a plurality of unspecified locations. Therefore, when the same object is detected in a plurality of unspecified locations (i.e., when location information corresponding to a plurality of unspecified locations is acquired for the same object), the registration unit 12 may register the object as a real object associated with the first user.

[0038] According to the fourth example of the registration process, an object that is likely to be owned by the first user can be registered as a real object associated with the first user based on location information. Therefore, when a setting is made to associate content with a real object, the content can be presented to the first user at an appropriate timing (frequency).

[0039] FIG. 2 is a diagram showing an example of a list (i.e., list information) of real objects registered by the registration unit 12. In this example, the registration information (record) corresponding to each real object includes an ID, item information, unique information, location information, detection frequency, and 3D information. The "ID" is identification information for uniquely identifying each real object. The "item information" is information for specifying an item (product) corresponding to a real object, and includes an item name (product name) and a model number. The "unique information" is unique information registered by the second example of the registration process described above. The "location information" is location information acquired in the fourth example of the registration process described above. The "detection frequency" is detection frequency acquired (calculated) in the third example of the registration process described above. The "3D information" is 3D data of the real object. In this example, for a real object with ID "001", the 3D data of the item acquired in the first example of the registration process described above (i.e., 3D data prepared in advance for the product corresponding to the real object) is registered as the 3D data of the real object. On the other hand, for the real objects with IDs "002" and "003", 3D data showing the entire object actually obtained by the recognition unit 21 or the detection unit 11 is registered as the 3D data of the real object. Note that in the example of Fig. 2, the real object with ID "002" which has a relatively low detection frequency of 0.1 times per hour is registered, but in the third example of the registration process described above, if the threshold value is set to "2 times / h", the real object with ID "002" will be excluded from the registration targets.

[0040] The list presentation unit 13 presents to the second user list information indicating a list of multiple real objects registered by the registration unit 12. For example, the list presentation unit 13 presents to the second user list information (see FIG. 2) stored in the list DB 30. In this embodiment, the list presentation unit 13 presents the list information to the second user by transmitting the list information to the operation terminal 40 of the second user. Note that the list presentation unit 13 may present to the second user only some of the items of the list information shown in FIG. 2 (for example, text information indicating item information and visually displayed 3D data).

[0041] The operation terminal 40 is, for example, any terminal such as a PC or a smartphone used by the second user. The operation terminal 40 has a list receiving unit 41, a list display unit 42, and an input unit 43. The list receiving unit 41 receives list information transmitted from the list presenting unit 13. The list display unit 42 displays the list information on a display unit such as a display provided in the operation terminal 40. The input unit 43 accepts an operation of the second user. More specifically, the input unit 43 accepts an operation of selecting one or more real objects X from among a plurality of real objects included in the list information displayed on the display unit. The input unit 43 also accepts an operation of specifying content Y to be associated with the real object X. In the present embodiment, as an example, the content Y is a content including a message of the second user. For example, the content Y is a spherical object (for example, an object resembling a soap bubble), and when a predetermined operation of the first user on the object (for example, an operation of holding a hand over the object) is performed, the message of the second user is displayed. An example of the content displayed on the display unit 22 of the HMD 20 will be described later.

[0042] The associating unit 14 associates virtual content that does not exist in the real space with the real object registered by the registration unit 12. In this embodiment, the associating unit 14 associates content Y designated by the second user with a real object X selected by the second user from among a plurality of real objects included in the list information. That is, the associating unit 14 receives information on the real object X and content Y selected and designated by the input unit 43 of the operation terminal 40 from the operation terminal 40, and associates the content Y with the real object X. Information indicating that the content Y is associated with the real object X is notified to the display control unit 15. Note that the correspondence between the real object and the content is not limited to a "1:1" relationship. That is, a plurality of contents may be associated with one real object, or a common content may be associated with a plurality of real objects. In this embodiment, as an example, it is assumed that one content Y is associated with one real object X.

[0043] The display control unit 15 controls the display of the display unit 22 of the HMD 20 to display the content on the display unit 22 together with the real space. More specifically, when a real object X is included in the real space displayed on the display unit 22, the display control unit 15 displays the content Y in association with the real object X. For example, the display control unit 15 transmits 3D data of the real object X and display information of the content Y (for example, information used for rendering) to the HMD 20 as display control information. The HMD 20 controls the display of the content Y on the display unit 22 based on the display control information. That is, the HMD 20 uses the 3D data of the real object X to determine whether or not an object corresponding to the real object X exists in the real space displayed on the display unit 22. Then, when an object corresponding to the real object X is detected in the real space displayed on the display unit 22, the HMD 20 displays the content Y in association with the object. For example, the HMD 20 displays the content Y on the display unit 22 at a position overlapping with the detected object or in the vicinity of the detected object. In addition, the determination of whether or not an object corresponding to real object X exists in the real space displayed on the display unit 22 may be performed by the HMD 20 as in the above example, or may be performed by the server 10 (display control unit 15).

[0044] FIG. 3 is a diagram showing an example of display of content. In this example, the real object X is a lighting fixture placed on a desk in the home of the first user. The content Y is a content including a message from the second user as described above. As shown in (A) of FIG. 3, according to the display control of the display control unit 15 described above, when an object corresponding to the real object X is detected in the real space projected on the display area 22a of the display unit 22, the content Y is displayed in association with the real object X. In this example, the content Y is initially displayed as an object resembling a soap bubble floating so as to overlap the real object X.

[0045] Next, when the first user wearing the HMD 20 executes a predetermined operation on the content Y (for example, an operation of holding a hand over the content Y), the HMD 20 or the display control unit 15 detects the predetermined operation and changes the state of the content Y. In this example, as shown in (B) of Fig. 3, the content Y changes to text information indicating a message set by the second user (in this example, "How are you?") after a soap bubble popping effect display is displayed.

[0046] Next, an example of the operation of the content providing system 1 (including the content providing method according to the embodiment) will be described with reference to FIG.

[0047] In step S1, the recognition unit 21 of the HMD 20 of the first user captures a real space image.

[0048] In step S2, the recognition unit 21 analyzes the real space image. For example, the recognition unit 21 detects an object included in the real space image by performing at least one of image recognition, photogrammetry, and 3D scanning. For example, the recognition unit 21 acquires (generates) 3D data of the object.

[0049] In step S3, the HMD 20 transmits to the server 10 the 3D data of the object obtained by the analysis, as well as the time information and the location information described above.

[0050] In step S4, the detection unit 11 of the server 10 detects a real object that is an object included in the real space image captured by the HMD 20. As an example, the detection unit 11 detects the real object by acquiring 3D data detected by the HMD 20 (recognition unit 21).

[0051] In step S5, the registration unit 12 registers the real object detected in step S4 in association with the first user. The registration unit 12 may, for example, execute the processes of the first and second examples of the registration process described above. The registration unit 12 may also execute the processes of the third and fourth examples of the registration process described above. An example of the processing procedure of the third and fourth examples of the registration process of the registration unit 12 will be described with reference to the flowchart shown in FIG. 5. Note that the process of the flowchart in FIG. 5 is executed individually for each detected object (real object).

[0052] In step S51, the registration unit 12 determines whether the detection frequency within a predetermined period of time of the real object that is a candidate for registration is equal to or greater than a threshold. If the detection frequency within the predetermined period of time is not equal to or greater than the threshold (step S51: NO), the registration unit 12 does not register the real object in association with the first user. On the other hand, if the detection frequency within the predetermined period of time is equal to or greater than the threshold (step S51: YES), the registration unit 12 further executes the determination in step S52.

[0053] In step S52, the registration unit 12 determines whether the detection location of the real object is the private space of the first user or a plurality of unspecified locations by referring to the location information of the real object that is a candidate for registration. If the determination result is YES, the registration unit 12 registers the real object in association with the first user (step S53). On the other hand, if the determination result is NO, the registration unit 12 does not register the real object in association with the first user.

[0054] 4, in step S6, the list presentation unit 13 presents to the second user list information indicating a list of the multiple real objects registered by the registration unit 12. That is, the list presentation unit 13 transmits the list information to the operation terminal 40. The list information is received by the list receiving unit 41 of the operation terminal 40.

[0055] In step S7, the list display unit 42 of the operation terminal 40 displays the list information.

[0056] In step S8, the input unit 43 of the operation terminal 40 accepts an input operation of the second user. That is, the input unit 43 acquires information on the real object X selected by the second user and information on the content Y to be associated with the real object X.

[0057] In step S9, information indicating the real object X and the content Y is transmitted from the operation terminal 40 to the server 10.

[0058] In step S10, the associating unit 14 associates the content Y with the real object X.

[0059] In step S11, the display control unit 15 controls the display of the display unit 22 of the HMD 20 to display content together with a real space on the display unit 22. For example, when a real object X is included in the real space displayed on the display unit 22, the display control unit 15 transmits display control information including 3D data of the real object X and information on the content Y to the HMD 20 so that the content Y is displayed in association with the real object X.

[0060] In step S12, the HMD 20 executes display control based on the display control information, and when an object corresponding to the real object X is detected in the real space displayed on the display unit 22, content Y is displayed in association with the object (see FIG. 3).

[0061] In the server 10 described above, an object included in a real space image captured by the HMD 20 of the first user is automatically registered as a real object associated with the first user. That is, an object present around the user when the first user uses the HMD 20 (in other words, an object that catches the first user's eye) can be automatically registered as a real object. Therefore, the first user does not need to manually register an object present around the user, and the first user's effort is reduced. Then, when a real object X is displayed on the display unit 22 of the HMD 20, virtual content Y is displayed in association with the real object X. According to the above mechanism, it is possible to display the virtual content Y in association with an object that is likely to be seen by the first user when using the HMD 20, and therefore the content Y can be effectively presented to the first user. As a result, the convenience of the first user in the AR experience or MR experience can be improved.

[0062] Furthermore, the detection unit 11 detects, as a real object, an object included in a real space image corresponding to the field of view of the first user captured by the HMD 20 worn on the head of the first user. According to the above configuration, an object that is visually recognized by the first user when the first user uses (wears) the HMD 20 (i.e., an object that is likely to be seen by the first user when using the HMD 20) can be registered as a real object associated with the first user. Therefore, when a setting is made to associate content with a real object, it becomes possible to present the content to the first user at an appropriate timing (frequency).

[0063] The server 10 also includes a list presenting unit 13 that presents list information indicating a list of a plurality of real objects (real objects associated with the first user) registered by the registration unit 12 to the second user. The association unit 14 also associates content Y designated by the second user with a real object X selected by the second user from among a plurality of real objects included in the list information. According to the above configuration, the second user can appropriately grasp a list of real objects associated with the first user (i.e., objects that are likely to be seen by the first user while the first user is wearing the HMD 20). As a result, the second user can associate content Y that the second user wants to show to the first user while the first user is wearing the HMD 20 with a real object X that is likely to be seen by the first user. As a result, a mechanism for presenting content from the second user to the first user by AR or MR can be appropriately realized. In the present embodiment, the content Y includes a message input by the second user. In this case, a mechanism for transmitting a message from the second user to the first user by AR or MR can be appropriately realized.

[0064] In the above embodiment, the content Y designated by the second user is associated with the real object X selected by the second user different from the first user. However, the real object X and the content Y may be selected and designated by the first user himself. In this case, the operation terminal 40 in the above embodiment may be configured as a terminal operated by the first user. According to such a configuration, for example, the first user can associate content including a memorandum or other note with a real object that is an arbitrary property of the first user. As a result, when the first user visually recognizes the real object while using the HMD 20, the first user can check the contents of the note created by the first user himself.

[0065] In the above embodiment, the server 10 functions as a content providing device, but the functions of the server 10 may be implemented in the HMD 20. In this case, the HMD 20 functions as a content providing device. Also, some of the functions of the server 10 described above may be executed by the HMD 20. In this case, a computer system including the server 10 and the HMD 20 functions as a virtual content providing device.

[0066] In the above embodiment (FIG. 5), a flow in which both the third and fourth examples of the registration process are executed is shown, but this flow is merely an example. That is, in step S5 of FIG. 4, only one of the third and fourth examples of the registration process may be executed, or neither the third and fourth examples of the registration process may be executed.

[0067] In the above embodiment, 3D data of an object included in a real space image is acquired, but it is not essential to use 3D data. That is, the process required for the above-mentioned display control (i.e., detection of an object included in the real space displayed on the display unit 22) may be executed based on a 2D image of the object (real object).

[0068] Moreover, the block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

[0069] Functions include, but are not limited to, judging, determining, assessing, calculating, processing, deriving, investigating, searching, verifying, receiving, sending, outputting, accessing, resolving, selecting, choosing, establishing, comparing, anticipating, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.

[0070] For example, the server 10 according to an embodiment of the present disclosure may function as a computer that performs the virtual content providing method of the present disclosure. Fig. 6 is a diagram showing an example of a hardware configuration of the server 10 according to an embodiment of the present disclosure. The server 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0071] In the following description, the term "apparatus" may be replaced with a circuit, a device, a unit, etc. The hardware configuration of the server 10 may be configured to include one or more of the devices shown in FIG. 6, or may be configured to exclude some of the devices. The HMD 20 may also have a hardware configuration similar to that of the server 10 shown in FIG. 6.

[0072] Each function of the server 10 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, so that the processor 1001 performs calculations, controls communications via a communication device 1004, and controls at least one of reading and writing of data in the memory 1002 and the storage 1003.

[0073] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like.

[0074] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the programs. As the programs, programs that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments are used. For example, each functional unit of the server 10 (e.g., the display control unit 15, etc.) may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be similarly realized. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunication line.

[0075] The memory 1002 is a computer-readable recording medium, and may be configured by at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software module, etc. that can be executed to implement a content providing method according to an embodiment of the present disclosure.

[0076] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0077] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0078] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0079] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0080] The server 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0081] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as a modified or altered form without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

[0082] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0083] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0084] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0085] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0086] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0087] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0088] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0089] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0090] The names used for the above-mentioned parameters are not limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not limiting in any way.

[0091] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0093] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0094] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0095] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]

[0096] 1...content providing system, 10...server (content providing device), 11...detection unit, 12...registration unit, 13...list presentation unit, 14...association unit, 15...display control unit, 20...HMD (user terminal), 22...display unit, X...real object, Y...content.

Claims

1. a detection unit that detects a real object that is an object included in a real space image captured by a user terminal used by a first user; a registration unit that registers the real object detected by the detection unit in association with the first user; an associating unit that associates virtual content that does not exist in a real space with the real object registered by the registration unit; a display control unit that displays the content together with a real space on a display unit of a user terminal used by the first user; a list presentation unit that presents list information indicating a list of the plurality of real objects registered by the registration unit to the first user or a second user different from the first user, when the real object is included in a real space displayed on the display unit, the display control unit displays the content in association with the real object; The list presentation unit presents the list information to the second user, A content providing device in which the association unit associates the content including a message input by the second user with the real-world object selected by the second user from among the multiple real-world objects included in the list information.

2. A detection unit that detects a real object that is an object included in a real space image captured by a user terminal used by a first user; a registration unit that registers the real object detected by the detection unit in association with the first user; an associating unit that associates virtual content that does not exist in a real space with the real object registered by the registration unit; a display control unit that displays the content together with a real space on a display unit of a user terminal used by the first user; when the real object is included in a real space displayed on the display unit, the display control unit displays the content in association with the real object; The registration unit identifies an item corresponding to the object by comparing 3D data corresponding to a portion of the object obtained from the real space image with 3D data of a plurality of items prepared in advance, and registers the 3D data of the identified item as information of the real object.

3. A detection unit that detects a real object that is an object included in a real space image captured by a user terminal used by a first user; a registration unit that registers the real object detected by the detection unit in association with the first user; an associating unit that associates virtual content that does not exist in a real space with the real object registered by the registration unit; a display control unit that displays the content together with a real space on a display unit of a user terminal used by the first user; when the real object is included in a real space displayed on the display unit, the display control unit displays the content in association with the real object; The registration unit registers the object, whose frequency of detection from the real space images taken within a specified period is equal to or greater than a predetermined threshold, as the real object associated with the first user.

4. A detection unit that detects a real object that is an object included in a real space image captured by a user terminal used by a first user; a registration unit that registers the real object detected by the detection unit in association with the first user; an associating unit that associates virtual content that does not exist in a real space with the real object registered by the registration unit; a display control unit that displays the content together with a real space on a display unit of a user terminal used by the first user; when the real object is included in a real space displayed on the display unit, the display control unit displays the content in association with the real object; The registration unit acquires location information indicating the location where the real-space image was taken, and determines whether to register an object detected from the real-space image as the real object associated with the first user based on the location information.

5. The content providing device of claim 4, wherein the registration unit registers the object detected from the real space image as the real object associated with the first user when the location information indicates a location registered as a private space of the first user.

6. The content providing device of Claim 4, wherein the registration unit registers the object as the real object associated with the first user when the location information corresponding to multiple unspecified locations is obtained for the same object.

7. the user terminal includes a head mounted display that is worn on a head of the first user, The detection unit detects, as the real object, an object included in the real space image corresponding to a field of view of the first user captured by the head mounted display. The content providing device according to any one of claims 1 to 6.

8. a list presentation unit that presents list information indicating a list of the plurality of real objects registered by the registration unit to the first user or a second user different from the first user; the associating unit associates the content designated by the first user or the second user with the real-world object selected by the first user or the second user from among the plurality of real-world objects included in the list information; The content providing device according to any one of claims 2 to 6.

9. The detection unit detects the real object based on a result of performing at least one of image recognition, photogrammetry, and 3D scanning on the real space image. The content providing device according to any one of claims 1 to 6.

10. the registration unit registers, as part of information about the real object, unique information indicating a characteristic unique to the object together with article information indicating an article corresponding to the object detected from the real space image; The content providing device according to any one of claims 1 to 6.

Images