Information processing device

Abstract

From among the items that can be added, identify the items that the user needs at that time. The present invention provides an information processing device, information processing system, and program that display a virtual screen. [Solution] Equipped with a processor, the processor acquires images of the captured real space and We obtain schedule information, which is information about the schedule, and based on the schedule information, we capture the real space. From among them, identify the target to which the virtual screen will be attached, and then attach the virtual screen to the target. An information processing device that controls the display of associated data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Conventionally, there is an information processing apparatus that can display and visually recognize a virtual screen superimposed on the real space. Specifically, an information processing apparatus that specifies an addition target from the real space and associates and displays a virtual screen with the addition target is known.

[0003] Patent Document 1 describes that an information processing apparatus displays a superimposed image in which a virtual object located in a virtual three-dimensional space corresponding to the real world is superimposed on an image of the real world captured by an imaging unit on a display. Based on the user's line of sight detected by the line-of-sight detection unit, the information processing apparatus selects a virtual object included in the superimposed image, and draws this virtual object closer to the user within the virtual three-dimensional space so that the virtual object is displayed within a range reachable by the user's hand in the virtual three-dimensional space.

[0004] Patent Document 2 describes that an information processing apparatus acquires a captured image of the real space captured by an imaging display device, and controls the imaging display device to display the captured image with an additional image superimposed thereon. Specifically, the information processing apparatus extracts information corresponding to a certain condition from an information group regarding a first user who uses, for example, the imaging display device, and superimposes and displays the extracted information at a position corresponding to the first user in the real space on an imaging display device used by, for example, a second user. At this time, the information group to be displayed is, for example, the attributes or actions of the first user or posts by the first user on SNS.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] When users utilize information processing devices as described above, if a virtual screen is associated with and displayed for every possible item that could be added, it is possible that not only the information the user needs, but also information they do not need, will be displayed. The present invention aims to provide an information processing device, an information processing system, and a program that identify the necessary additional elements for the user at that time from among the elements that can be added and display a virtual screen. [Means for solving the problem]

[0007] The invention described in claim 1 is an information processing device comprising a processor, wherein the processor acquires an image of a captured real space, acquires schedule information which is information about the user's schedule, identifies an object to be added to from the captured real space based on the schedule information, and performs control to associate the virtual screen with the object to be added and display it. The invention described in claim 2 is an information processing device according to claim 1, wherein the added object is a use object which is an object used by a person and / or user identified based on the planned information. The invention described in claim 3 is an information processing device according to claim 2, wherein the planned information is information that has been agreed upon to meet a person when the attached object is a person, and information that has been agreed upon with the owner of the used object to use when the attached object is a used object. The invention described in claim 4 is an information processing device according to claim 2, wherein the planned information, when the attached object is a person, includes information including meeting the person at a predetermined time, and when the attached object is a usage object, includes information including using the usage object at a predetermined time. The invention described in claim 5 is an information processing device according to claim 1, wherein the object to be added is identified as a position in the real space, and the virtual screen is added adjacent to the identified position. The invention described in claim 6 is an information processing device according to claim 5, wherein the position of the added object is determined based on characteristic points of the added object included in an image captured from real space. The invention described in claim 7 is an information processing device according to claim 6, wherein the feature is the face of a person the user meets. The invention described in claim 8 is an information processing device according to claim 5, wherein the position of the added object is determined by a position determination means. The invention described in claim 9 is an information processing device according to claim 1, wherein the virtual screen is displayed within a predetermined time range and hidden outside of that time range. The invention described in claim 10 is the information processing apparatus according to claim 9, wherein the time range includes the scheduled start time and / or end time. The invention described in claim 11 is an information processing device according to claim 9, wherein the virtual screen is hidden from view when a predetermined time has elapsed from the scheduled start time. The invention described in claim 12 is an information processing device according to claim 9, wherein the virtual screen is hidden from view when the added object continues to be recognized for a predetermined period of time. The invention described in claim 13 is an information processing device according to claim 9, wherein the virtual screen is not hidden when there are multiple added objects, but maintains a displayed state. The invention described in claim 14 is an information processing device according to claim 1, wherein the virtual screen is hidden from view when the added object comes within a predetermined distance. The invention described in claim 15 is an information processing device according to claim 1, wherein the virtual screen is hidden when the user is deemed to have memorized the added object. The invention described in claim 16 is an information processing device according to claim 1, wherein the virtual screen is the content included in the scheduled information. The invention described in claim 17 is an information processing device according to claim 16, wherein the contents included in the scheduled information are the contents necessary to identify a person to be identified based on the scheduled information and / or the contents necessary to identify a user object that is to be used by the user. The invention described in claim 18 is an information processing device according to claim 16, wherein the virtual screen is changed after a predetermined time has elapsed based on the scheduled information. The invention described in claim 19 is an information processing system comprising: a management device for managing schedule information, which is information relating to a user's schedule; a display means for displaying a virtual screen superimposed on real space; an imaging means for imaging real space; and a processor, wherein the processor of the information processing system displays a virtual screen based on the schedule information managed by the management device, and the processor of the information processing system acquires an image of real space from the imaging means, acquires the schedule information from the management device, identifies an attachment target from the real space imaged by the imaging means based on the schedule information, associates the virtual screen with the attachment target, and controls the display means to display it. The invention described in claim 20 is a program for a computer that enables the following functions: acquiring an image of a captured real space; acquiring schedule information which is information relating to the user's schedule; identifying an object to be added to from the captured real space based on the schedule information, which is an object to which a virtual screen will be added; and performing control to associate the virtual screen with the object to be added and display it. [Effects of the Invention]

[0008] According to the invention of claim 1, it is possible to provide an information processing device that identifies an additional object necessary for the user at that time from among the possible additional objects and displays a virtual screen. According to the invention of claim 2, information necessary for the user to identify the person they are meeting and / or the object of use can be displayed as a virtual screen. According to the invention of claim 3, when the schedule is certain by agreement, the necessary information can be displayed as a virtual screen, and when it is uncertain, it can be not displayed. According to the invention of claim 4, when the schedule is certain because the time is fixed, the necessary information can be displayed as a virtual screen, and when it is uncertain, it can be not displayed. According to the invention of claim 5, it becomes easier for the user to recognize the object to be added. According to the invention of claim 6, it becomes easier to identify the object to which the addition will be made. According to the invention of claim 7, it becomes easier to identify a person as the object to be attached. According to the invention of claim 8, the location of a person can be determined even without knowing the characteristic points of the object to be added. According to the invention of claim 9, information can be displayed according to the user's schedule. According to the invention of claim 10, a virtual screen can be displayed according to the user's schedule. According to the invention of claim 11, the virtual screen can be hidden when it is considered that the user's schedule has started. According to the invention of claim 12, the virtual screen can be hidden when it is considered that the user has recognized the object to be added. According to the invention of claim 13, it is possible to avoid the user confusing multiple attachments. According to the invention of claim 14, the virtual screen can be hidden when it is considered that the user has recognized the object to be added. According to the invention of claim 15, unnecessary information can be prevented from being displayed as a virtual screen. According to the invention of claim 16, it becomes easier for the user to keep track of their schedule. According to the invention of claim 17, it becomes easier for the user to determine whether the object to be added is a person or an object of use. According to the invention of claim 18, necessary information can be displayed as a virtual screen according to the user's situation. According to the invention of claim 19, an information processing system can be provided that identifies an addition target necessary for the user at that time from among those that can be addition targets and displays a virtual screen. According to the invention of claim 20, a function can be realized by a computer that can provide a function of identifying an addition target necessary for the user at that time from among those that can be addition targets and displaying a virtual screen.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram showing an information processing system to which the present embodiment is applied. [Figure 2] It is a diagram showing the configuration of the AR glasses to which the present embodiment is applied. [Figure 3] It is a diagram showing the optical system of the AR glasses. [Figure 4] It is a flowchart explaining the operation of the AR glasses. [Figure 5] (a) to (c) are diagrams showing Display Example 1. [Figure 6] (a) to (c) are diagrams showing Display Example 2. [Figure 7] It is a diagram showing Display Example 3. [Figure 8] It is a diagram showing Display Example 4. [Figure 9] It is a diagram showing Display Example 5. [Figure 10] It is a diagram showing Display Example 6. [Figure 11] It is a flowchart explaining the operation of the AR glasses when controlling the time for displaying a virtual screen. [Figure 12] (a) to (d) are diagrams showing the setting of a predetermined time when displaying a virtual screen.

Embodiments for Carrying Out the Invention

[0010] <Configuration of Information Processing System 1> Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram showing an information processing system 1 to which the present embodiment is applied. The illustrated information processing system 1 is configured by connecting an AR glass 10, which is an example of an information processing device, and a management server 20, which is an example of a management device, via a network 30.

[0011] <Configuration of AR Glass 10> FIG. 2 is a diagram showing the configuration of the AR glass 10 to which the present embodiment is applied. The AR glass 10 is a so-called glasses-type wearable terminal device. Here, "wearable" means that it can be worn by a user. Therefore, a glasses-type wearable terminal device is a computer device in the form of glasses that can actually be worn on the head by the user. Since the AR glass 10 is glasses-type, it includes a lens unit 104, a bridge 105, and temple pieces 106. Note that the "lens unit" is the name of the component of the lens unit 104 in the glasses-type wearable terminal device, and includes things that are not lenses. That is, since the AR glass 10 is glasses-type, for convenience, the expression "lens unit" is used, but it does not actually need to have the function of a lens. That is, the lens unit 104 does not need to have an optical function of refracting light.

[0012] The AR glass 10 is a device that realizes AR display for the user. Here, "AR" refers to "Augmented Reality", which means displaying a virtual screen superimposed on the real space for the user. That is, the user can view the virtual screen through the AR glass 10 and can also view the real space through the lens unit 104 of the AR glass 10. Also, the "virtual screen" is the display of an image created by a computer and viewable using a device such as the AR glass 10, and the "real space" is the actually existing space.

[0013] The content displayed on the virtual screen is not particularly limited. For example, it could include exercise data such as steps taken, calories burned, and distance traveled; health data such as the user's heart rate and blood pressure; notification data such as incoming emails and messages; and navigation data such as maps and guides of the surrounding area. However, as will be explained in more detail later, the content displayed on the virtual screen will be related to the user's schedule information. Here, AR glasses 10 are shown as a so-called glasses-type device, but the shape and format of any device that displays AR to the user are not particularly limited. For example, it could be an information terminal device such as a smartphone or tablet.

[0014] Figure 3 shows the optical system of the AR glasses 10. Figure 3 illustrates the AR glasses 10 as viewed from direction III in Figure 2. In Figure 3, when the user wears the AR glasses 10, the component on the left is denoted with "L", and the component on the right is denoted with "R". Various methods exist for the AR glasses 10 that realize the display of a virtual screen, such as virtual image projection and retinal projection, and any of these methods is acceptable. The AR glasses 10 have, for example, the following configuration. These AR glasses 10 use the retinal projection method.

[0015] The AR glasses 10 include laser light sources 101L and 101R, optical fibers 102L and 102R, mirrors 103L and 103R, lens sections 104L and 104R, a bridge 105, temples 106L and 106R, cameras 107L and 107R, a communication module 108, and a control module 109.

[0016] The laser light sources 101L and 101R are light sources for creating virtual screens. By using three lasers—Red, Green, and Blue—and switching between them at high speed, the laser light sources 101L and 101R can create full-color virtual screens.

[0017] Optical fibers 102L and 102R are provided inside temples 106L and 106R, respectively, and guide the laser light La emitted from laser light sources 101L and 101R to mirrors 103L and 103R. Optical fibers 102L and 102R may be made of glass or plastic.

[0018] Mirrors 103L and 103R reflect the laser beam La at an angle that is almost perpendicular to its direction of propagation, guiding it to the lens units 104L and 104R. These mirrors 103L and 103R can pivot up, down, left, and right, which changes the angle at which the laser beam enters the lens units 104L and 104R. As a result, the position at which the laser beam reaches the user's retina (ML and MR) also changes up, down, left, and right. Consequently, a two-dimensional image can be viewed by the user as a virtual screen.

[0019] The lens sections 104L and 104R are examples of two lenses and consist of light-transmitting members positioned in the user's line of sight when worn on the user's head. Here, "light-transmitting member" refers to a member that transmits visible light. Note that the light-transmitting member does not need to transmit all visible light. That is, the light-transmitting member may have the function of a so-called filter, transmitting only a portion of the wavelengths in the visible light region. The lens sections 104L and 104R each have a light-guiding section 114L and 114R and a reflecting section 124L and 124R inside, respectively. The light-guiding sections 114L and 114R guide the laser beam La, whose angle has been changed by the mirrors 103L and 103R, towards the bridge 105 by totally reflecting it. The reflective sections 124L and 124R reflect the respective laser beams La guided by the light guide sections 114L and 114R in nearly right-angle directions, changing the direction of the laser beams La towards the retina ML of the user's left eye and the retina MR of the user's right eye. Furthermore, since the lens sections 104L and 104R are light-transmitting materials, the user can view the real world through the lens sections 104L and 104R. This allows the user to view the virtual screen superimposed on the real world.

[0020] The bridge 105 rests on the user's nose to support the AR glasses 10 and is a component for attaching the AR glasses 10 to the user's head. Temples 106L and 106R are components that support the AR glasses 10 by being placed on the user's ears and are used to attach the AR glasses 10 to the user's head. Cameras 107L and 107R capture images in the direction that is in front of the user. Cameras 107L and 107R are examples of imaging means for capturing images of real space.

[0021] The communication module 108 is a communication interface for communicating with the outside world. The control module 109 controls the operation of the laser light sources 101L and 101R and the mirrors 103L and 103R when displaying a virtual screen. The control module 109 can be realized by using a CPU, internal memory, and external memory to run control software that controls the laser light sources 101L and 101R and the mirrors 103L and 103R. Here, the CPU is an example of a processor. In this embodiment, the laser light sources 201L and 201R, the optical fibers 202L and 202R, the mirrors 203L and 203R, and the lens units 204L and 204R function as display means for displaying a virtual screen superimposed on the real space for the user.

[0022] <Configuration of Management Server 20 and Network 30> The management server 20 manages schedule information, which is information regarding the user's schedule. The management server 20 includes a CPU (Central Processing Unit) as arithmetic means, a main memory as storage means, and a storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Here, the CPU executes various software such as an OS (basic software) and application software (application software). The main memory is a storage area for storing various software and data used for its execution, and the storage is a storage area for storing input data for various software and output data from various software. Furthermore, the management server 20 includes a communication interface for communicating with the outside (hereinafter referred to as "communication I / F"), a display device including a video memory and a display, and input devices such as a keyboard, a mouse, and a touch panel.

[0023] The network 30 is a communication means used for information communication between the AR glasses 10 and the management server 20, and is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network). The communication line used for data communication may be wired or wireless, or a combination of these. Also, the AR glasses 10 and the management server 20 may be connected via a relay device such as a gateway device or a router through a plurality of networks and communication lines.

[0024] In the information processing system 1, the AR glasses 10 acquire information regarding the user's schedule from the management server 20 via the network 30. Then, the AR glasses 10 create and display the display content of the virtual screen based on this information.

[0025] <Operation of the AR glasses 10> Next, the operation of the AR glasses 10 will be described in detail. FIG. 4 is a flowchart for explaining the operation of the AR glasses 10. First, the user turns on the AR glasses 10 (step 101). This activates the various mechanisms of the AR glasses 10. Next, the AR glasses 10 retrieve schedule information from the management server 20 (step 102). "Schedule information" is information about the user's schedule. Schedule information includes appointments to meet people and appointments for the user to use objects. Examples of appointments to meet people include meeting appointments, hospital appointments, and dinner reservations. Examples of appointments to use objects include car rental reservations, conference room reservations, and seat reservations for transportation such as trains and airplanes. More specifically, in the case of appointments to meet people, the schedule information includes the name of the person to meet, the location of the meeting, and the start and end times of the meeting. In the case of appointments to use objects, the schedule information includes the name of the object to be used, the identification number of the object, the location of the object, and the start and end times of using the object. Examples of object identification numbers include car license plates, conference room numbers, and vehicle seat numbers.

[0026] Furthermore, if the scheduled information involves a meeting with a person, it may also include information indicating that the meeting with that person has been agreed upon. In this case, the scheduled information includes information that will be met with the person at a predetermined time. Furthermore, if the scheduled information concerns the use of an object, it may also include information indicating that the use of the object has been agreed upon with the owner of the object. In this case, the scheduled information includes information that the object will be used at a predetermined time. When a meeting schedule has been approved through groupware or similar systems, it is assumed that there is an agreement to meet the person. Similarly, when a reservation has been confirmed through a facility reservation system, it is assumed that there is an agreement to use the equipment in question. The schedule information is obtained by the communication module 108 of the AR glasses 10 from the management server 20 via the network 30.

[0027] Next, the AR glasses 10 use cameras 107L and 107R to capture images of the user's surroundings (step 103). This can also be described as the AR glasses 10 using cameras 107L and 107R to capture images of the real world. Furthermore, the control module 109 acquires the captured image of the real space (step 104). Furthermore, the control module 109 identifies the object to be added from the captured real space based on the scheduled information (step 105). The "object to be added" is the object to which the virtual screen is added. In other words, the control module 109 controls the display of a virtual screen on the object to be added that exists in the real space. The object to be added is a person and / or an object used by the user, identified based on the scheduled information. That is, the object to be added is a person the user will meet or an object the user will use. For example, if the scheduled information is that the user will meet a person, then that person is the object to be added. Also, if the scheduled information is that the user will use an object, then that object is the object to be added. In this case, the object to be used may be movable property or immovable property.

[0028] Next, the control module 109 creates information for a virtual screen to be displayed superimposed on the real space, based on the scheduled information (step 106). As will be explained in more detail later, this virtual screen is determined from the information that can be displayed based on the scheduled information and is identified as necessary for the user. Then, the control module 109 performs control to associate and display a virtual screen with the attached object (step 107). In this case, the virtual screen to be displayed will be the information required by the user, as identified in step 106. The control module 109 controls the laser light sources 101L, 101R and the mirrors 103L, 103R, as described in Figure 3, to perform control to display this virtual screen. After that, the process from step 101 is repeated at predetermined intervals.

[0029] <Example Display> Next, we will explain a specific example of the content displayed on the virtual screen shown in step 107 of Figure 4. (Example 1) Figures 5(a) to 5(c) show an example of display 1. Example 1 shows a scenario where a user has an appointment to meet someone for the first time. Figure 5(a) shows the schedule information. In this case, the schedule information includes the start and end times of the scheduled event, the content of the event, the location of the event, and the attendees. Specifically, the schedule information includes "13:00~14:00" as the start and end times, "Meeting" as the content of the event, "10F Free Space" as the location of the event, and "Attendee: Hanako" as the attendee. This schedule information is entered in advance by a user using the AR glasses 10, for example, using a terminal device such as a PC or smartphone. This schedule information is then stored on the management server 20.

[0030] Figure 5(b) shows information about the schedule stored on the management server 20. In this case, the management server 20 stores the name, face, and ID information of the attendee "Hanako" included in the schedule information. This information is entered, for example, by someone who has met the attendee "Hanako" in the past, and then stored by the management server 20. The information shown in Figures 5(a) and 5(b) is sent to the AR glasses 10, as explained in step 102 of Figure 4.

[0031] Figure 5(c) shows the real-world space and virtual screen that the user actually sees through the AR glasses 10. As shown in the diagram, a dotted circular object Ob1 is displayed as a virtual screen over the face of the attendee, "Hanako". In addition, some of the schedule information is displayed as object Ob2, and an arrow object Ob3 is displayed to link objects Ob1 and Ob2. In this case, the object to be added is the attendee, "Hanako." The object to be added is identified as a location in real space based on the schedule information. This location of the object to be added is identified based on the feature points of the object to be added that are included in the image captured in real space. In this case, the feature points are the faces of the people the user will meet. In this case, the location of "Hanako" is identified based on the facial information of the attendee, "Hanako," which is stored by the management server 20. Then, a virtual screen is added adjacent to the location identified as the target for attachment. In this case, objects Ob1 to Ob3 will be displayed adjacent to the attendee "Hanako". In Figure 5, facial recognition is used to identify the location of the attendee "Hanako," and virtual screen objects Ob1 to Ob3 are added adjacent to it.

[0032] (Example 2) Figures 6(a) to 6(c) show the second example of the display. Display Example 2, like Display Example 1, shows a scenario where the user has an appointment to meet someone for the first time. Of these, Figure 6(a) shows the schedule information. In this case, the schedule information includes facial information of the attendee "Hanako," compared to the case shown in Figure 5(a). Figure 6(b) shows the schedule information stored on the management server 20. This information, as in Figure 5(b), includes the name, face, and ID of the attendee, "Hanako". Figure 6(c) shows the real-world space and virtual screen that the user actually sees through the AR glasses 10. As shown in the diagram, a dotted rectangular object Ob4 is displayed as a virtual screen on the attendee's "Hanako" ID card. In addition, some of the schedule information is displayed as object Ob5, and an arrow object Ob6 is displayed to link objects Ob4 and Ob5.

[0033] In this case as well, the object to be attached is the attendee, "Hanako." On the other hand, the location of the object to be attached is determined by a location identification means. The "location identification means" is a device or item used to determine the location of the object to be attached. In this case, the location identification means is the ID card held by the attendee, "Hanako." In this case, the location of the ID card is determined by, for example, a beacon. The location of the ID card is then acquired by the AR glasses 10, and the location of the attendee, "Hanako," who is the object to be attached, is determined. Then, a virtual screen is attached adjacent to the location determined as the object to be attached. In this case, objects Ob4 to Ob6 are displayed adjacent to the attendee, "Hanako." In this case, the face of attendee "Hanako" stored on the management server 20 may be from the past and may differ from the current face. In this case, "Hanako" cannot be identified by face recognition, but she can be identified by using the ID card, which is a means of location identification. Furthermore, if the information written on the ID card is recognizable, the attendee "Hanako" can also be identified by the ID written on the ID card. Additionally, a new face image can be fed back from the AR glasses 10 to the management server 20, and the information related to the schedule on the management server 20 can be updated with the new face image.

[0034] (Example 3) Figure 7 shows an example of display example 3. Example 3 shows a virtual screen that will be displayed during a meeting that the user has scheduled. Here, object Ob7 is displayed as a virtual screen adjacent to attendee Ua of the other party conducting the meeting. Object Ob7 is the business card image of attendee Ua. Similarly, object Ob8 is displayed as a virtual screen adjacent to attendee Ub of the other party conducting the meeting. Object Ob8 is the business card image of attendee Ub. In this case, the objects to be attached are attendees Ua and Ub. In addition to business card images, the names, phonetic spellings, departments, job titles, and facial images of the other party's attendees Ua and Ub may also be displayed. These business card images are stored, for example, on the management server 20, and the AR glasses 10 retrieve them from the management server 20 based on the schedule information.

[0035] (Example 4) Figure 8 shows an example of display example 4. Example 4 shows a virtual screen displayed for a vehicle that a user plans to use. This vehicle can be reserved and used by the user through services such as car sharing or car rental. In this case, the scheduled information includes, for example, the start and end times of use of the vehicle to be used, the make and model, license plate number, color, size, exterior image, and location. The vehicle is the target of the user's use and the target to which the virtual screen is attached. The AR glasses 10 then create and display a virtual screen based on the scheduled information. In this case, the location of the vehicle used by the user is indicated by objects Ob9 and Ob10. Object Ob9 displays "Your Car" to indicate that it is a vehicle reserved by the user, and object Ob10 is an arrow that links the vehicle to object Ob9. The location of the vehicle can be determined, for example, by parking the vehicle in a predetermined parking spot. It can also be determined by the vehicle's onboard GPS (Global Positioning System) unit. This GPS unit can be considered an example of a location determination method.

[0036] (Example 5) Figure 9 shows an example of display example 5. Display Example 5, like Display Example 4, shows a virtual screen that is displayed for a vehicle that the user plans to use. However, this example shows a different virtual screen from the one in Figure 8. In this case, the location of the car the user will be using is indicated by objects Ob11 and Ob12. Object Ob11 displays "Your Car Reservation No. 22123," indicating that it is the car the user has reserved and showing the reservation number. Object Ob12 is an arrow that links the car to object Ob11. In addition, the car reservation details are displayed as object Ob13. Here, in addition to the title "XX Car Share," the reservation number, reservation time, vehicle, and estimated price are displayed.

[0037] (Example 6) Figure 10 is a diagram illustrating display example 6. Example 6 shows a virtual screen displayed for a booth that a user has scheduled to use. This booth can be reserved and used by the user, for example, by subscribing to a service such as a shared office. In this case, the scheduled information would include, for example, the start and end times of use, the location, and the booth number of the booth to be used. Furthermore, the target of the user's interaction with the virtual screen would be this booth. In this case, objects Ob14 and Ob15 indicate the location of Booth Ba, which the user will be using, out of Booths Ba and Bb. Object Ob14 displays "This is your booth" to indicate that it is the booth the user has reserved, as well as the location of the booth and the start and end times of use. Object Ob15 is also displayed as an arrow linking Booth Ba and Object Ob14.

[0038] Furthermore, the virtual screen includes the content contained in the schedule information. This allows the user to check the schedule information. In other words, the content contained in the schedule information functions as the information necessary to identify the person being identified based on the schedule information. It also functions as the information necessary to identify the target user who will be using it. Furthermore, the virtual screen may be configured to change after a predetermined time has elapsed, based on the schedule information. This ensures that the virtual screen displays information that is useful at that time. For example, before the start of a meeting, information identifying individuals who are meeting for the first time, as shown in Display Examples 1 and 2, may be displayed, and after the start of the meeting, information such as the names and titles of the individuals, as shown in Display Example 3, may be displayed. Alternatively, the timing of the change may be set to when the target information has been recognized for a predetermined period of time. That is, when the target information has been recognized for a predetermined period of time, it is assumed that the user already knows the people they are scheduled to meet and the target information they are scheduled to use, so the information is switched to other information.

[0039] <Controlling the display time of the virtual screen> Next, we will explain how to control the display time of a virtual screen. Specifically, the virtual screen is displayed within a predetermined time range, and is hidden outside of this time range. Figure 11 is a flowchart illustrating the operation of the AR glasses 10 when controlling the time the virtual screen is displayed. The flowchart shown in Figure 11 differs from the flowchart shown in Figure 4 in that steps 201-205 and 208-209 are the same as steps 101-105 and 106-107, with the addition of steps 206-207. Therefore, the following explanation will focus on the processing of steps 206-207.

[0040] In step 206, the control module 109 determines whether or not the predetermined time for displaying the virtual screen has arrived (step 206). If the predetermined time has not yet arrived (No in step 206), the control module 109 does not create or display a virtual screen (step 207). On the other hand, if the time is predetermined (Yes in step 206), the control module 109 creates information for a virtual screen to be displayed superimposed on the real space based on the scheduled information (step 208), and performs control to display it (step 209).

[0041] The following describes the predetermined time for Step 206. Figures 12(a) to (d) illustrate the predetermined time settings for displaying a virtual screen. The time range shown in the illustration includes the scheduled start time and / or end time. Figure 12(a) shows a case where the virtual screen is displayed for a period of time from the start time to the end time of the user's scheduled appointment. That is, the virtual screen is displayed when the scheduled start time arrives. Then, when the scheduled end time arrives, the virtual screen is hidden from view. This is an example of controlling the display and hiding of the virtual screen according to the user's scheduled time.

[0042] Furthermore, Figure 12(b) shows a case where a virtual screen is displayed within a time range that includes the start and end times of the user's scheduled appointment, specifically from a predetermined time before the start time to a predetermined time after the end time. In other words, the virtual screen is displayed starting a little before the scheduled start time. Also, the virtual screen is hidden a little after the scheduled end time. This ensures that necessary information is presented to the user within a certain time range, both before the start time and after the end time.

[0043] On the other hand, in Figure 12(c), the virtual screen is displayed starting a predetermined time before the user's scheduled start time. The virtual screen is then hidden once the predetermined time has elapsed from the scheduled start time. In other words, for example, as in display examples 1 and 2 explained in Figures 5 and 6, if the user has an appointment to meet someone for the first time, it is more convenient for the user to display the screens in Figures 5 and 6 before the scheduled start time. Also, once the scheduled start time has passed, it is expected that the user has already met this person, so there is no problem in hiding the virtual screens shown in display examples 1 and 2.

[0044] Furthermore, as shown in Figure 12(c), the timing for hiding the virtual screen may be set so that it is hidden when the target object is recognized for a predetermined period of time. In other words, for example, as in Display Examples 1 and 2 explained in Figures 5 and 6, if a user has an appointment to meet someone for the first time, it is expected that the user has already met this person when the target object is recognized for a predetermined period of time, so there is no problem in hiding the virtual screen shown in Display Examples 1 and 2.

[0045] Furthermore, as shown in Figure 12(c), the timing for hiding the virtual screen may be set so that it is hidden from its displayed state when the target object comes within a predetermined distance. In other words, for example, as in Display Examples 1 and 2 explained in Figures 5 and 6, if a user has an appointment to meet someone for the first time, when this person, who is the target object, comes within a predetermined distance, it is expected that the user has already recognized this person and is meeting them, so there is no problem in hiding the virtual screen shown in Display Examples 1 and 2.

[0046] Furthermore, it is possible to intentionally hide information at the time the virtual screen is displayed. This is represented by a dotted line in Figure 12(d). For example, if the user is considered to have memorized the information to be added, it will be hidden. For instance, if the user has already met this person more than a predetermined number of times, it is assumed that the user has memorized the person's face, etc. In this case, it will be hidden even at the time the virtual screen is displayed. This prevents unnecessary information from being displayed as a virtual screen.

[0047] Furthermore, when there are multiple attachment targets, the virtual screen may remain displayed rather than being hidden. For example, in a meeting as shown in Display Example 3 in Figure 7, if there are multiple attendees from the other party as attachment targets, the virtual screen will not be hidden. This helps to prevent users from confusing multiple attachment targets.

[0048] In the example described above, the display information for the virtual screen was created by the AR glasses 10 based on the schedule information, but it may also be created on the management server 20, sent to the AR glasses 10, and then displayed.

[0049] <Program Description> In this embodiment, the processing performed by the AR glasses 10 described above is carried out by a program such as control software.

[0050] Therefore, the program that enables the processing performed by the AR glasses 10 in this embodiment can be understood as a program that enables the computer to perform the following functions: acquiring images of the captured real space; acquiring schedule information, which is information about the user's schedule; identifying an object to be added to from the captured real space based on the schedule information; and performing control to associate the virtual screen with the object to be added and display it.

[0051] Furthermore, the program that implements this embodiment can be provided not only by communication means, but also by being stored on a recording medium such as a CD-ROM.

[0052] Although this embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear from the claims that various modifications or improvements made to the above embodiment are also included in the technical scope of the present invention. [Explanation of Symbols]

[0053] 1…Information processing system, 10…AR glasses, 20…Management server, 101L, 101R…Laser light source, 102L, 102R…Optical fiber, 103L, 103R…Mirror, 104, 104L, 104R…Lens section, 106L, 106R…Temple, 107L, 107R…Camera, 109…Control module

Claims

1. Equipped with a processor, The aforementioned processor, Acquire images of the captured real space, Retrieve the schedule information, which is information about the user's schedule. Based on the aforementioned planned information, the object to which a virtual screen will be added is identified from the captured real space. An information processing device that performs control for associating and displaying the virtual screen with the aforementioned attachment target.

2. The information processing apparatus according to claim 1, wherein the added object is a use object which is an object used by a person and / or user identified based on the scheduled information.

3. The information processing apparatus according to claim 2, wherein the aforementioned scheduled information is information that an agreement has been reached to meet a person when the attached object is a person, and information that an agreement has been reached with the owner of the use object to use the attached object when the attached object is a use object.

4. The information processing apparatus according to claim 2, wherein the aforementioned scheduled information, when the attached object is a person, includes information including meeting the person at a predetermined time, and when the attached object is an object for use, includes information including using the object for use at a predetermined time.

5. The information processing device according to claim 1, wherein the object to be added is identified as a location in the real space, and the virtual screen is added adjacent to the identified location.

6. The information processing apparatus according to claim 5, wherein the position of the object to be added is determined based on the characteristic points of the object to be added that are included in an image captured from real space.

7. The information processing apparatus according to claim 6, wherein the aforementioned feature is the face of the person the user meets.

8. The information processing apparatus according to claim 5, wherein the position of the added object is determined by a position determination means.

9. The information processing apparatus according to claim 1, wherein the virtual screen is displayed within a predetermined time range and is hidden outside of that time range.

10. The information processing apparatus according to claim 9, wherein the time range includes the scheduled start time and / or end time.

11. The information processing apparatus according to claim 9, wherein the virtual screen is hidden from view when a predetermined time has elapsed from the scheduled start time.

12. The information processing apparatus according to claim 9, wherein the virtual screen is hidden from view when the added object continues to be recognized for a predetermined period of time.

13. The information processing apparatus according to claim 9, wherein the virtual screen is not hidden when there are multiple attachment targets, but maintains a displayed state.

14. The information processing apparatus according to claim 1, wherein the virtual screen is hidden from display when the object to be added comes within a predetermined distance.

15. The information processing apparatus according to claim 1, wherein the virtual screen is hidden when the user is deemed to have memorized the added object.

16. The information processing apparatus according to claim 1, wherein the virtual screen is the content included in the scheduled information.

17. The information processing apparatus according to claim 16, wherein the contents included in the aforementioned scheduled information are the contents necessary to identify a person to be identified based on the scheduled information and / or the contents necessary to identify the object to be used by the user.

18. The information processing device according to claim 16, wherein the virtual screen is changed after a predetermined time has elapsed based on the schedule information.

19. A management device that manages schedule information, which is information related to the user's schedule, An information processing device comprising: display means for displaying a virtual screen superimposed on real space; imaging means for capturing images of real space; and a processor, which displays a virtual screen based on the schedule information managed by the management device, Equipped with, The processor of the information processing device is An image of the real space is obtained from the aforementioned imaging means, The schedule information is obtained from the management device. Based on the aforementioned planned information, the object to which a virtual screen will be added is identified from the real space captured by the imaging means, An information processing system that controls the association of the virtual screen with the added object and displays it on the display means.

20. On the computer, A function to acquire images of the captured real space, A function to retrieve schedule information, which is information about the user's schedule, Based on the aforementioned planned information, the function identifies the object to which a virtual screen will be added from within the captured real space, A function for controlling the display of the virtual screen associated with the attached object, A program to achieve this.

Images