Information processing device and program

Abstract

To present information to a user after making a determination on the presentation of the information using schedule information of the user.SOLUTION: A CPU identifies the presence of a coffee shop S around a user. The CPU makes a determination on presentation, to the user, of information regarding a coffee shop D, which is in predetermined relationship with the coffee shop S in advance, the coffee shop D being located in a location different from that of the coffee shop S, on the basis of schedule information of the user.SELECTED DRAWING: Figure 23

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Patent document 1 discloses a process in which, when a user's activity state is determined to be a first state, a first operation input method is selected for operating an operation target with a first operation, and, when the user's activity state is determined to be a second state, a second operation input method is selected for operating the operation target with a second operation different from the first operation. Patent Document 2 discloses a process for notifying a user of stored schedule data when the time from the current time to the start time of an action reaches a predetermined time. Patent Document 3 discloses a process of generating an itinerary from a departure point to a destination, and causing a display control unit and a first display unit to display route guidance based on the itinerary. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-120550 [Patent Document 2] Japanese Patent Application Laid-Open No. 2002-108829 [Patent Document 3] Japanese Patent Application Laid-Open No. 2008-77311 Summary of the Invention [Problem to be solved by the invention]

[0004] By presenting the user with information about objects that exist around the user and objects that exist in different locations from the objects, the user can learn about the objects that exist around the user and objects that exist in different locations from the objects. In this case, simply presenting information that is associated in advance with an object to users would result in common information being uniformly presented to each user. It is an object of the present invention to use a user's schedule information to make decisions regarding the presentation of information about an object before presenting information about the object to the user. [Means for solving the problem]

[0005] The invention of claim 1 is an information processing device having a processor, the processor comprising: Based on a video showing the user's surroundings, or based on an electronic map, the user's current location, and the direction the user is facing, Identifying the presence of a first object in the vicinity of a user, and based on schedule information that is information about the schedule of the user, The existence was identified The information processing device makes a decision regarding presentation of first object information, which is information about the first object, to the user, and the first object is an object not included in the schedule information. In the invention described in claim 2, the processor performs the following steps based on the schedule information: Deciding whether to present the first object information to the user and / or determining the content of the first object information to be presented to the user. The information processing device according to claim 1. The invention described in claim 3 is an information processing device described in claim 1, wherein the processor controls the first item information to include campaign information for the first item when the schedule information satisfies predetermined conditions. The invention of claim 4 is an information processing device having a processor, the processor comprising: Based on the image showing the user's surroundings or an electronic map, the user's current location, and the direction the user is facing, Identifying the presence of a second object in the vicinity of the user, and based on schedule information that is information about the schedule of the user, The existence was identified An information processing device that makes a decision regarding the presentation of first object information to the user about a first object that has a predetermined relationship with the second object and exists in a location different from the second object, and the first object is an object that is not included in the schedule information. The invention described in claim 5 is an information processing device described in claim 1 or 4, wherein the processor controls the first item information to include information about a campaign for the first item when the schedule information satisfies predetermined conditions, and controls the first item information not to include information about a campaign for the first item when the schedule information does not satisfy predetermined conditions. The invention described in claim 6 is an information processing device described in claim 4, in which the processor controls so that when second object information about the second object present within the user's field of view is presented to the user, the first object information is also presented to the user. The invention described in claim 7 is an information processing device described in claim 1, in which the processor controls the first object information to be presented to the user so that it does not overlap the first object that is visible to the user. The invention described in claim 8 is an information processing device described in claim 4, in which the processor controls the first object information to be presented to the user so that it does not overlap the second object visible to the user. The invention described in claim 9 is the information processing device described in claim 4, wherein the processor makes a decision regarding presentation of first object information about the first object to the user. The invention described in claim 10 is an information processing device described in claim 6, in which the processor controls the second item information to be presented to the user as campaign information for the second item, which becomes valid only when the first item is used. An invention recited in claim 11 is the information processing device recited in claim 6, wherein the second object information presented to the user prompts the user to make a reservation for a service provided by the second object. The invention described in claim 12 is an information processing device described in claim 4, in which the processor controls the presentation of information about a campaign for the first item, which becomes valid only when the second item is used, as the first item information. The invention of claim 13 is the information processing device of claim 1 or 4, wherein the first object information presented to the user prompts the user to make a reservation for a service provided by the first object. The invention described in claim 14 is an information processing device described in claim 4, in which the processor, when multiple objects are associated with the second object, determines the first object from among the multiple objects based on status information of the second object. The invention described in claim 15 is an information processing device described in claim 4, in which the processor determines the first object from among the multiple objects based on status information of each of the multiple objects when multiple objects are associated with the second object. The invention described in claim 16 is an information processing device described in claim 4, wherein the processor, when multiple objects are associated with the second object, determines the first object from among the multiple objects based on the user's level of interest in each of the multiple objects. The invention described in claim 17 is characterized in that the processor acquiring, as the schedule information, information about the location and start time of the next schedule of the user who passed by the first object; The aforementioned schedule information the location and start time of the next appointment acquired as the first item, and the next reservation start time of the first item. When the user uses the first item based on the in Benefit or disadvantage and the benefit or disadvantage to the provider of the first item. Determine Decided The said advantages or disadvantages beneficially and making a decision regarding the presentation of the first object information to the user based on the 1 to The information processing device is described. An invention recited in claim 18 is the information processing device recited in claim 17, wherein the processor determines a predicted usage time of the first object of the user as the benefit or disadvantage. The invention described in claim 19 is an information processing device described in claim 1 or 4, wherein the schedule information includes a type of schedule of the user, and the processor makes a decision regarding the presentation of the first object information to the user based on the type. The invention described in claim 20 is an information processing device described in claim 1 or 4, wherein the schedule information includes information of people related to the user's schedule, and the processor makes a decision regarding the presentation of the first object information to the user based on the number of people. The invention described in claim 21 is an information processing device described in claim 1 or 4, wherein the processor acquires at least the location and start time of the user's scheduled event as the schedule information, and makes a decision regarding the presentation of the first object information to the user based on the remaining time from the current time to the start time and the travel time required to travel from the current location to the event location. The invention described in claim 22 is an information processing device described in claim 1 or 4, wherein the processor makes a decision regarding the presentation of the first object information to the user based on the acquired schedule information and status information of the first object. The invention described in claim 23 is an information processing device described in claim 4, wherein the processor makes a decision regarding the presentation of the first object information to the user based on the acquired schedule information and status information of the second object. The invention described in claim 24 is an information processing device described in claim 1 or 4, in which the information processing device identifies objects present around the user based on images obtained by a camera moving with the user. The invention described in claim 25 is an information processing device described in claim 1 or 4, wherein the information processing device is a mobile device that moves with the user, and the processor identifies objects present in the user's vicinity based on the user's current location and the direction the user is facing. The invention described in claim 26 is the information processing device described in claim 1 or 4, wherein the information processing device is a device worn in front of the eyeballs of the user. The invention described in claim 27 is an information processing device described in claim 26, wherein the information processing device has a see-through display, and the processor associates the first object information with a real object in real space that the user views through the display. In the invention described in claim 28, the processor acquires information about the schedule of each of a plurality of users present around the first object as the schedule information, and selects a user from among the plurality of users based on the schedule information. Identify users who meet certain conditions, and then The first object information is presented to Ruyu -The asThe information processing device according to claim 1, wherein the information processing device determines whether or not the first and second parameters are the same. In the invention described in claim 29, the processor acquires information about the schedule of each of the plurality of users present around the second object as the schedule information, and selects a user from among the plurality of users based on the schedule information. Identify users who meet certain conditions, and then The destination to which the first object information is presented Naruyu -The as The information processing device according to claim 4, wherein the information processing device determines the The invention described in claim 30 is an information processing device described in claim 1 or 4, wherein the information processing device communicates with a mobile device that moves with the user, and the processor controls the first object information to be displayed on the display of the mobile device. The invention described in claim 31 is Based on a video showing the user's surroundings, or based on an electronic map, the user's current location, and the direction the user is facing, Based on a function of identifying the presence of a first object in the vicinity of a user and schedule information that is information about the user's schedule, The existence was identified a function of making a decision regarding presentation of first object information, which is information about the first object, to the user; The first entity is a program, which is an entity not included in the schedule information. The invention described in claim 32 is Based on the image showing the user's surroundings or an electronic map, the user's current location, and the direction the user is facing, Based on a function of identifying the presence of a second object in the vicinity of the user and schedule information that is information about the user's schedule, The existence was identified and a function to make a decision regarding the presentation to the user of first object information about a first object that has a predetermined relationship with the second object and exists in a location different from the second object, wherein the first object is an object not included in the scheduled information. [Effects of the Invention]

[0006] According to the invention of claim 1, it is possible to use the user's schedule information to make a decision regarding the presentation of information about an object, and then present the information about the object to the user. According to the invention of claim 2, based on the schedule information, Decide whether to present the first object information to the user and decide the content of the first object information to be presented to the user. It is possible. According to the invention of claim 3, when the schedule information satisfies a predetermined condition, the first item information can include campaign information for the first item. According to the invention of claim 4, it is possible to use the user's schedule information to make a decision regarding the presentation of information about an object, and then present the information about the object to the user. According to the invention of claim 5, if the scheduled information satisfies a predetermined condition, the first item information can be configured to include information about the campaign for the first item, and if the scheduled information does not satisfy the predetermined condition, the first item information can be configured not to include information about the campaign for the first item. According to the invention of claim 6, when second object information about a second object present within the user's field of view is presented to the user, first object information can also be presented to the user. According to the invention of claim 7, the first object information can be presented to the user so as not to overlap with the first object visible to the user. According to the invention of claim 8, the first object information can be presented to the user so as not to overlap with the second object visible to the user. According to the invention of claim 9, after a decision is made regarding the presentation of first object information about the first object to the user, the first object information can be presented to the user. According to the invention of claim 10, it is possible to present to the user information about a campaign for the second item that becomes valid on the condition that the first item is used. According to the invention of claim 11, it is possible to present information that encourages a user to make a reservation for a service provided by the second object as the second object information presented to the user. According to the invention of claim 12, it is possible to present, as the first item information, information about a campaign for the first item that becomes effective on the condition that the second item is used. According to the invention of claim 13, the first object information presented to the user can be information that encourages a reservation for a service provided by the first object. According to the invention of claim 14, when a plurality of objects are associated with a second object, the first object can be determined from among the plurality of objects based on the situation information of the second object. According to the invention of claim 15, when a plurality of objects are associated with a second object, the first object can be determined from among the plurality of objects based on the status information of each of the plurality of objects. According to the invention of claim 16, when a plurality of objects are associated with a second object, the first object can be determined from among the plurality of objects based on the user's level of interest in each of the plurality of objects. According to the invention of claim 17, when the user uses the first object Benefits benefit or disadvantage and the benefit or disadvantage to the provider of the first item. Based on this, a decision can be made regarding the presentation of the first object information to the user. According to the invention of claim 18, a decision can be made regarding the presentation of the first object information to the user based on the predicted usage time of the first object by the user. According to the invention of claim 19, a decision can be made regarding the presentation of the first object information to the user based on the type of the user's schedule. According to the invention of claim 20, a decision can be made regarding the presentation of the first object information to the user based on information about people related to the user's schedule. According to the invention of claim 21, a decision can be made regarding the presentation of the first object information to the user based on the remaining time from the current time to the start time of the event and the travel time required to travel from the current location to the scheduled event location. According to the invention of claim 22, a decision can be made regarding the presentation of the first object information to the user based on the user's schedule information and the situation information of the first object. According to the invention of claim 23, a decision can be made regarding the presentation of the first object information to the user based on the user's schedule information and the situation information of the second object. According to the invention of claim 24, objects existing around the user can be identified based on images obtained by a camera that moves with the user. According to the invention of claim 25, objects existing around the user can be identified based on the current location of the user and the direction in which the user is facing. According to the invention of claim 26, the information processing device can be a device that is worn in front of the user's eyeballs. According to the invention of claim 27, the first object information can be associated with a real object in real space visually recognized by the user via the display. According to the invention of claim 28, among a plurality of users present around the first object, Meet certain conditions The first item information can be presented to the user. According to the invention of claim 29, among a plurality of users present around the second object, Meet certain conditions The first item information can be presented to the user. According to the invention of claim 30, the first object information can be displayed on a mobile device that moves with the user. According to the invention of claim 31, it is possible to use the user's schedule information to make a decision regarding the presentation of information about an object, and then present the information about the object to the user. According to the invention of claim 32, it is possible to use the user's schedule information to make a decision regarding the presentation of information about an object, and then present the information about the object to the user. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] 1 shows an example of the hardware configuration of an external server. [Figure 3] FIG. 10 is a diagram illustrating a display screen displayed on a display of a user's terminal device. [Figure 4] FIG. 10 is a diagram illustrating a display screen displayed on a display of a user's terminal device. [Figure 5] FIG. 10 is a diagram illustrating a display screen displayed on a display of a user's terminal device. [Figure 6] FIG. 10 is a diagram illustrating the display levels of surrounding object information. [Figure 7] FIG. 10 is a diagram illustrating the display levels of surrounding object information. [Figure 8] FIG. 10 is a diagram illustrating the display levels of surrounding object information. [Figure 9] FIG. 10 is a diagram showing another example of display on the display screen of the terminal device. [Figure 10] FIG. 10 is a diagram showing another example of display on the display screen of the terminal device. [Figure 11] FIG. 10 is a diagram showing another example of display on the display screen of the terminal device. [Figure 12] FIG. 10 is a diagram showing another example of display on the user's terminal device. [Figure 13] FIG. 10 is a diagram showing another example of display on the user's terminal device. [Figure 14] FIG. 2 is a diagram showing the hardware configuration of a terminal device. [Figure 15] 1 is a flowchart showing an example of a flow of processing executed in the present embodiment. [Figure 16] 10 is a flowchart illustrating another example of processing. [Figure 17] 10 is a flowchart illustrating another example of processing. [Figure 18] 10 is a flowchart illustrating another example of processing. [Figure 19] 10 is a flowchart illustrating another example of processing. [Figure 20] FIG. 1 is a diagram showing a user's travel route. [Figure 21] FIG. 1 is a diagram showing a user's travel route. [Figure 22] FIG. 10 is a diagram showing another example of display on the terminal device. [Figure 23] FIG. 10 is a diagram showing another example of display on the terminal device. [Figure 24-1] FIG. 10 is a diagram showing another example of display on the terminal device. [Figure 24-2] FIG. 10 is a diagram showing another example of display on the terminal device. [Figure 25] FIG. 10 is a diagram showing another example of display on the terminal device. [Figure 26]FIG. 10 is a diagram showing a display example in which a user who has time to spare until his or her next appointment that is equal to or greater than a predetermined threshold has walked up to a nearby object. [Figure 27] 10 is another example of display on the terminal device. [Figure 28] 10 is a flowchart showing a processing flow. [Figure 29] FIG. 10 is a diagram showing a database referred to by a CPU. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram showing an example of the configuration of an information processing system 1 according to this embodiment. In this embodiment, a plurality of information processing devices cooperate as a system, specifically, a terminal device 200 owned by a user and an external server 100 that communicates information with the terminal device 200 are provided. In this embodiment, the terminal device 200 and the external server 100 are connected via a communication line such as an Internet line. In the example shown in Fig. 1, a glasses-type terminal device 200A and a wrist-worn type terminal device 200B are worn by a user as examples of terminal devices 200. Note that, although the terminal device 200 is worn by the user in Fig. 1, the terminal device 200 is not limited to a terminal device worn by a user. For example, the terminal device 200 may be a smartphone type. Furthermore, the number of terminal devices 200 owned by one user does not need to be two, and may be one, or three or more. Although not shown in FIG. 1, a terminal device 200 is provided corresponding to each of a plurality of users, and in this embodiment, the terminal devices 200 of the plurality of users are connected to the external server 100.

[0009] FIG. 2 shows an example of the hardware configuration of the external server 100. As shown in FIG. The external server 100 (an example of an information processing device) is provided with an information processing unit 101, an information storage device 102 that stores information, and a network interface 103 that realizes communication via a LAN (Local Area Network) cable or the like.

[0010] The information processing unit 101 has a CPU (=Central Processing Unit) 111 as an example of a processor, a ROM (=Read Only Memory) 112 in which basic software, a BIOS (=Basic Input Output System), etc. are stored, and a RAM (=Random Access Memory) 113 used as a work area. The CPU 111 may be a multi-core CPU, and the ROM 112 may be a rewritable non-volatile semiconductor memory. The information storage device 102 is realized by an existing information storage device such as a hard disk drive, a semiconductor memory, or a magnetic tape. The information processing unit 101, the information storage device 102, and the network interface 103 are connected via a bus 106 and a signal line (not shown).

[0011] The program executed by CPU 111 may be provided to external server 100 in a state where it is stored in a computer-readable recording medium such as a magnetic recording medium (such as a magnetic tape or a magnetic disk), an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory. Also, the program executed by CPU 111 may be provided to external server 100 using a communication means such as the Internet.

[0012] The external server 100 communicates with a terminal device 200 (see FIG. 1) that moves with the user. The CPU 111 of the external server 100 controls information to be presented to the user. As an example, the external server 100 controls the presentation of information about surrounding objects located around the user (hereinafter, also referred to as "surrounding object information"). The CPU 111 of the external server 100 identifies surrounding objects around the user based on an image captured by a camera 214 (see FIG. 1) (for example, a camera 214 provided in the terminal device 200) that moves with the user. Note that the identification of surrounding objects may be performed by a device other than the external server 100, for example, the terminal device 200.

[0013] The CPU 111 of the external server 100 controls the presentation of information about the identified surrounding object. More specifically, the CPU 111 of the external server 100 controls the display of the surrounding object information on the display 217 of the terminal device 200 (see FIG. 1).

[0014] In this embodiment, the case where the presentation of information on the terminal device 200 is controlled by the external server 100 will be described. However, the control may be performed by one of the terminal devices 200 without using the external server 100. Even when the information to be presented to the user is controlled by a device other than the external server 100, various information such as the user's schedule information may be stored in the external server 100. In this case, the terminal device 200 acquires information about the user such as schedule information by accessing the external server 100, and determines the information to be presented to the user based on the acquired information. Furthermore, the method of presenting information is not limited to visual methods such as displaying on a display, but may be presented by voice, for example.

[0015] In this embodiment, schedule information, which is information about the schedule of a user who owns the terminal device 200, is stored in the information storage device 102 (see FIG. 2) of the external server 100. Then, the CPU 111 of the external server 100 determines information to be presented to the user based on this schedule information. Then, when the information determined to be presented is transmitted to the terminal device 200, the information is presented on the terminal device 200. Here, a plan refers to something that is scheduled or has been planned. Planned events can include not only events that actually took place as planned, but also events that did not take place as planned. Examples of plans include meetings, business trips, classes, exams, travel, meals, accommodation, meeting people, shopping, going to the hospital, going to the hair salon, picking up children from kindergarten, and doing homework. Plans do not have to be special or out of the ordinary.

[0016] Examples of schedule information include the schedule name, type, location, scheduled time (start time, end time, deadline, etc.), whether it is set to repeat, related people (attendees, organizers, etc.), notification settings, attachments, related URLs, a map of the venue, and notes. Sources of schedule information include (i) a calendar, (ii) a communication history, and (iii) a history of a reservation system or a search service. Note that (i) to (iii) may be stored in the information storage device 102 or in other information storage devices. The CPU 111 may request and acquire schedule information from the stored information or history whenever necessary, or may acquire it in advance at a predetermined timing.

[0017] Examples of calendars in (i) include calendars provided by a calendar application installed on the user's terminal device and calendars provided by cloud services. Examples of calendars also include itineraries. A user's schedule may be registered by the user himself / herself, or may be automatically extracted and registered from other applications or linked services. If a user makes their calendar public or shares it with others, the user's schedule may also be registered by others. Furthermore, calendars used by medical institutions and the like may manage the visit schedules of multiple users.

[0018] Examples of communication history in (ii) include audio and video recordings of voice and video calls, emails, conversational messaging applications, social networking sites, etc. Examples of communication include visual communication using text, pictures, images, videos, etc., auditory communication using voice, etc., and combinations thereof. To acquire schedule information from communication history, appropriate technologies such as voice recognition, image analysis, and language analysis are used. For example, a history of the following series of messages exchanged between the user and person A in a conversational messaging application is stored in the information storage device of the user's smartphone or in the information storage device (not shown) of the conversational messaging application provider. User: "I'm going to Tokyo on July 5th for the first time in a while. Would you like to meet up if it's convenient for you?" Person A: "I'd love to meet you." User: "What's the best place and time?" Person A: "I'm taking the Shinkansen, so it would be great if we could meet near Tokyo Station. Would around 1:00 PM be convenient for you?" User: "Okay, let's meet at Tokyo Station at 1 PM."

[0019] Examples of the history of reservation systems and search services in (iii) include reservation history in reservation systems for hotels, restaurants, rental offices, conference rooms, seminars, concerts, and transportation facilities, as well as search history for route search services. From the search history of route search services, schedule information such as departure point, departure time, destination, arrival time, and means of transportation (walking, train, car, etc.) can be obtained. Regarding the method of acquiring the schedule information, the information in (i) to (iii) may be used as it is, or the results of processing such as voice recognition, image analysis, language analysis, etc. may be used as the schedule information. For example, in the example of (ii) of the interaction between the user and person A, this history is analyzed and the schedule information of the user, "Schedule ID: 001 (can be any ID), Name: Meet with person A, Start time: 1 PM, Location: Tokyo Station, Attendees: person A", is acquired.

[0020] In this embodiment, not only the user's schedule information but also actual information is sequentially acquired and accumulated. This allows information on the user's current actions, past actions, current state, and past state to be obtained. The actual information of the user is a general term for information on the user's actual actions and information on the user's actual state. Information about the actions actually taken by the user and the actual state of the user can also be managed by linking it to the date and time by adding a timestamp.

[0021] Sources of actual information include, for example, (a) location information, (b) communication history, (c) purchase and electronic payment history, (d) schedule completion information, (e) vital signs information, (f) visual information, (g) auditory information, and (h) other information managed by various applications and services.

[0022] (a) Location information Location information can be obtained based on GPS, traffic history, the origin of calls and live videos, and location information attached to social media posts and photos posted on social media. Traffic history also includes entry and exit history. Specific examples of traffic history include the entry and exit history at station ticket gates, and the history of passing through gates installed in facilities or rooms.

[0023] (b) Communication history The communication history can be the same as that described above in (ii). As an example, if a user says "I'm currently in front of the central ticket gate at Tokyo Station" during a call, actual information that the user was in front of the central ticket gate at Tokyo Station when this call was taking place can be obtained. As another example, if a user sends a message on a conversational messaging service saying "The baseball game just finished. It was fun," actual information that the user was watching the baseball game can be obtained. As another example, if a user posts a photo of a hamburger on a social networking site with the text "It was delicious," actual information that the user ate a hamburger can be obtained. As another example, from the participation status and participation history of an online conference, actual information that the user is in a conference or that the user attended a conference can be obtained.

[0024] (c) Purchase and electronic payment history Examples of electronic payment usage history include usage history of credit cards, transportation IC cards, code payments (such as QR Code (registered trademark) and barcode payments), etc.

[0025] Purchase history can be obtained based on the store's (either physical or online) sales history database, the information registered in the household management application, and the results of scanning receipts and invoices.

[0026] (d) Planned Completion Information Examples of schedule completion information include the completion status of a to-do list in which the user has registered things to do, or a shopping list in which the user has registered shopping plans. Specifically, for example, if the status of an item registered in a to-do list changes from incomplete to completed, it can be determined that the item (e.g., "Send Mom a Mother's Day present") has been carried out, and actual information can be obtained. Also, once the items on the shopping list are checked, it can be determined that the items have actually been purchased, and actual information can be obtained. If there is a function for managing whether or not a schedule registered in a calendar has actually been carried out, the information can be obtained based on the completion information of the schedule in the calendar.

[0027] (e) Vital Information Examples of vital information include information about heart rate, blood pressure, breathing, body temperature, blood sugar level, oxygen level, state of consciousness, brain waves, and pupil response. As an example, this information is measured by a sensor in a device worn by the user. As another example, information measured by a device (such as a thermometer or blood pressure monitor) installed in a city or hospital may be received and used.

[0028] (f) Visual information Visual information is information about what the user can see. What the user is looking at can be identified, for example, by analyzing image data (which may be a still image or a video) taken in the direction ahead of the user. More specifically, what the user is looking at can be identified by analyzing image data obtained by a camera that takes images in the direction of travel of the vehicle (e.g., a car) in which the user is riding. What the user is looking at can also be identified by analyzing image data obtained by a camera installed in a glasses-type terminal device that takes images in the direction the user's face is facing.

[0029] In addition, what the user is looking at can also be identified by analyzing image data obtained by cameras installed on roads or in stores that capture the situation in front of the user. Furthermore, for example, by analyzing image data obtained by a camera attached to a wearable terminal device attached to the user's head or eyeballs, it is possible to identify what the user is looking at.

[0030] Additionally, what a user is looking at can also be identified by analyzing images and videos taken with the camera on the smartphone the user is carrying. What the user is looking at can also be determined from information about the user's current location, the direction the user is facing, and an electronic map. What the user is looking at can also be determined by detecting the user's line of sight and what is in front of them.

[0031] (g) Auditory information The auditory information is information about sounds heard by the user, and is acquired by a microphone at the user's current location or a microphone provided in the terminal device 200 owned by the user.

[0032] (h) Other information managed by various applications and services The information managed by various applications and services is information managed by applications installed on the user's terminal device and services used by the user. One example is physical information managed by a health management application or service. Examples of physical information include height, weight, body fat percentage, BMI, etc. Physical information may be automatically collected when the user steps on a scale, or may be manually entered by the user. Another example is an exercise record managed by an exercise management application or service, where the exercise information may be automatically recognized or manually entered by the user. Another example is dietary information and nutritional information managed by a dietary management application or service. Some dietary management applications or services, for example, acquire information such as calories and nutrients from a photograph of food eaten (either uncooked or cooked). Alternatively, a user may manually input information about food eaten and ingested nutrients (calories, sugar content, salt content, etc.). Other examples include usage history of applications and services such as games, music, videos, maps, online meetings, videos, and e-books. Information about songs played by the user, information about songs given stars, and browsing and revision history of web browsers and file viewers can also be used.

[0033] In this embodiment, as will be described later, information is acquired about, for example, the user's level of hurry, the user's interest in surrounding objects, and the user's state. Here, the degree of urgency of the user can be determined based on, for example, schedule information, actual information, or a combination of these. As an example, the user's level of urgency can be determined based on the time required to travel from the user's current location (an example of actual information) to the scheduled event location (an example of scheduled information) and the time remaining from the current time until the scheduled start time (an example of scheduled information). As another example, the degree of hurry of the user can be determined based on the user's moving speed (an example of actual information) or the user's vital signs (an example of actual information). If the user's moving speed is faster than usual or if their heart rate or blood pressure is elevated, it can be considered that the user is in a hurry. As another example, the degree of hurry of a user can be determined by analyzing image data (an example of actual information) obtained by a camera provided on a wearable terminal device worn by the user. For example, if a user is checking the clock more frequently, it can be assumed that the user is concerned about time and is in a hurry.

[0034] A user's interest in the surroundings can be determined based on, for example, schedule information, actual information, and combinations thereof. For example, if a user has made a restaurant reservation (an example of schedule information) within a predetermined time from the current time, it can be determined that the user has little interest in eating at other restaurants along the way to that restaurant. As another example, if the user has an appointment in the calendar to meet Person A (an example of appointment information), it is possible that the user may give a present to Person A, and therefore it is possible to identify that the user may be interested in stores that sell items that could be used as presents for Person A before meeting Person A. Furthermore, if it is identified that the user has purchased a present for Person A (an example of actual information), it can be determined that the user has lost interest in other stores that sell items that could be used as presents for Person A.

[0035] As another example, if a user has a meal planned (an example of schedule information), it can be determined that the user may not be interested in eating at a restaurant until a certain period of time has passed since the meal was planned. As another example, if a user had a busy schedule last week (an example of schedule information), it can be determined that the user is likely tired. As another example, if a user has a long business trip planned for next week by plane (an example of schedule information), it can be determined that the user is likely interested in travel items. As another example, if a hospital appointment (an example of schedule information) is scheduled within a predetermined period from the current time (which may be before, after, or both before and after the current time), it can be determined that the user may have a health problem.

[0036] Other examples include whether a user is interested in an object, and the object of the user's interest, which can be determined based on information such as the user's vital signs, such as heart rate, the number of times the user has visited the object, the amount of time spent at the object, purchase history, whether the object is added to favorites, statements such as "I want XX," frequently opened applications, frequently opened websites, frequently listened to songs, and reactions on social media (such as giving a high rating to a particular post) (all of which are examples of real information).

[0037] As other examples, the user's condition can be determined based on information such as whether or not they often listen to soothing music, the amount of sleep they get, whether or not they are detected as dozing, the amount of caffeinated beverages they consume, vital signs, posts about their health on social media, information obtained by an exercise management application, whether or not the user has an appetite (for example, information about appetite determined by information obtained by a diet management application), weight gain or loss, history at hospitals or pharmacies, and prescription history (all of which are examples of actual information).

[0038] In this embodiment, the scheduled information and actual information are stored in the information storage device 102 of the external server 100, but the scheduled information and actual information may also be stored in the information storage device of the terminal device 200 or in the information storage device of another system to which the external server 100 or the terminal device 200 can be connected. Furthermore, the schedule information and actual information do not need to be stored in only one location, but may be stored in a distributed manner across multiple servers or terminal devices.

[0039] In this embodiment, schedule information of a user carrying terminal device 200 (see FIG. 1 ) is used to select an object for which information should be presented to the user from images captured by camera 214 provided on terminal device 200. Then, information about the selected object is presented to the user. More preferably, information about the selected object is presented in association with the object. Examples of presenting information in association with an object include displaying information superimposed on or near the object when the user is looking at the object, or displaying an arrow or speech bubble indicating the association between the object and the information. There are existing technologies that provide information about objects in captured video, but most of them only provide information about all recognized objects or present the same information to all users. Different users do not necessarily react in the same way to the same information, and while some users appreciate the information, there are also users who do not want the information or who find it annoying when it is presented to them. In this embodiment, in view of these circumstances, the presentation of information is controlled. Note that the various functions executed by the CPU 111 may be executed by a processor of the terminal device 200 that moves with the user, for example, by the CPU 211A of the terminal device 200.

[0040] 3 to 5 are diagrams illustrating examples of display screens displayed on the display 217 of the user's terminal device 200. The display screens shown in Figs. 3 to 5 are displayed under different conditions. In this embodiment, the display screen shown in FIG. 3 is displayed when the user's level of hurry satisfies a predetermined first condition and the road ahead of the user is a road the user has traveled less than a predetermined number of times (for example, a road the user has never traveled before). An example of a case in which the user's level of hurry satisfies the predetermined first condition is when an indicator indicating the user's time margin satisfies a predetermined condition. An example of an indicator indicating the user's time margin is the difference between the time remaining from the user's current time until the start time of the next event and the time required for the user to travel from the user's current location to the venue of the next event. The larger this difference, the greater the user's time margin. Another example of a case in which the user's level of hurry satisfies the predetermined first condition is when it can be determined that the user is running toward the venue of the next event. It can be determined that the user is running, for example, when the user's moving speed and heart rate are each equal to or greater than a predetermined threshold. It can also be determined that the user is running, for example, by analyzing video captured by the camera 214. When the user is moving in a hurry, the speed at which things change in the image captured by camera 214 becomes faster than when the user is walking normally. Furthermore, when the user is running, the image captured by camera 214 shakes up and down more than when the user is walking. By detecting the degree of this shaking, it is possible to identify that the user is running. Note that the predetermined first condition is assumed to indicate a higher degree of urgency than the predetermined second condition described below. Furthermore, the degree of urgency may be whether or not the user is in a hurry, or may be the degree of hurry, or may be the degree of non-urgency (i.e., the degree of leisure).

[0041] The display screen shown in Fig. 3 does not provide guidance on surrounding objects around the user. In other words, surrounding object information, which is information about surrounding objects around the user, is not displayed. In this embodiment, when the degree of hurry of the user satisfies the first condition, the CPU 111 determines not to present the surrounding object information to the user.

[0042] 3, a colored belt-shaped image 92 corresponding to the road is displayed, and the user is guided to the road that should be taken. The road that the user should take is, for example, the road that the user should take to reach the destination. In this embodiment, the CPU 111 determines to present route guidance information, which is information that guides the user from their current location to their destination, when the road in front of the user is a road that the user has traveled less than a predetermined number of times.

[0043] Note that the CPU 111 may determine to present route guidance information, which is information that guides the user from the current location to the destination, when the user's hurry satisfies the first condition, regardless of the number of times the user has traveled the road in front of the user. By doing so, it is expected that the user will proceed along the route without making detours.

[0044] In this embodiment, the display screen shown in Figure 4 is displayed when the user's level of urgency satisfies a predetermined second condition that is different from the first condition, and the road ahead of the user is a road that the user has traveled a predetermined number of times or more. The display screen shown in Fig. 4 does not provide the route guidance shown in Fig. 3. In this embodiment, the CPU 111 determines not to present route guidance information, which is information that guides the user from the current location to the destination, when the road ahead of the user is a road that the user has traveled a predetermined number of times or more. 4, information about surrounding objects located around the user is displayed and guidance about the surrounding objects is provided, as indicated by reference numerals 4A and 4B. In this embodiment, when the degree of hurry of the user satisfies a predetermined second condition different from the first condition, the CPU 111 determines to present the surrounding object information to the user.

[0045] In this embodiment, the camera 214 (see FIG. 1) of the terminal device 200 is provided so as to capture an image in the direction in which the face of the user wearing the terminal device 200 is facing. Then, the CPU 111 identifies one or more peripheral objects existing around the user based on the image captured by the camera 214 moving with the user. In other words, the CPU 111 identifies peripheral objects existing around the user based on the user's current location and the direction in which the user is facing. An example of peripheral object information is peripheral facility information, which is information about peripheral facilities existing around the user. It is not necessary to use a camera to identify objects around the user; instead, objects around the user located in the direction the user is facing can be identified based on the user's current location, the direction the user is facing, and an electronic map. The current location of the user can be identified by a GPS provided in the terminal device 200, and the direction in which the user is facing can be identified based on a direction sensor provided in the terminal device 200.

[0046] In this embodiment, the display screen shown in Figure 5 is displayed when the user's level of urgency satisfies a predetermined second condition that is different from the first condition, and the road ahead of the user is not a road that the user frequently travels. The display screen shown in Figure 5 displays surrounding object information, which is information about surrounding objects located around the user, as shown by symbols 5A and 5B, and also displays a strip-shaped image 92, which guides the user on the path they should take, as shown by symbol 5C.

[0047] 6 to 8 are diagrams for explaining the display amount levels of surrounding object information. The display of surrounding object information can be controlled not only by switching between enabled display, which allows the display to be performed, and disabled display, which prevents the display from being performed, but also by setting a display amount level based on scheduled information, and displaying the information on the display screen according to this determined display amount level. In this embodiment, the display screen shown in FIG. 6 is displayed to a moving user whose level of urgency satisfies a first condition. The first condition is satisfied when the level of urgency is higher than the second and third conditions, which will be described later. In this case, the CPU 111 sets, for example, the display amount level of surrounding object information to 0. The display screen shown in FIG. 6 does not display surrounding object information. On the other hand, the display screen shown in FIG. 6 displays route guidance. If route guidance is displayed when the user is in a high level of urgency, it is more likely that the user will proceed along the route without making detours.

[0048] In this embodiment, the display screen shown in Fig. 7 is displayed to a user who is traveling and whose level of urgency satisfies a second condition. The second condition is satisfied when the level of urgency is lower than the first condition and higher than a third condition, which will be introduced later. In this case, the CPU 111 sets, for example, the display amount level of the surrounding object information to 1. In this case, the surrounding object information is displayed as indicated by the reference numeral 7A.

[0049] In this embodiment, the display screen shown in Fig. 8 is displayed to a user who is traveling and whose level of urgency satisfies a third condition. The third condition is satisfied when the level of urgency is lower than the first and second conditions. In this case, in this embodiment, the CPU 111 sets the display level of the surrounding object information to 2. In this case, as shown by the reference numerals 8A and 8B, the surrounding object information is displayed in two places, and more surrounding object information is displayed compared to the display in FIG.

[0050] In this embodiment, the CPU 111 as an example of a processor controls the amount of surrounding object information to be presented to the user based on the schedule information. In addition, the amount of surrounding object information can be controlled by controlling the number of surrounding objects for which surrounding object information is displayed, or by controlling the amount of surrounding object information displayed for one surrounding object.

[0051] In the example described above, the case where the surrounding object information itself is displayed was described as an example, but this is not limited to this, and the surrounding object information may be displayed first as an icon corresponding to the surrounding object. When the user operates this icon, the surrounding object information as described above may be displayed.

[0052] Note that the user's operation on the icon may be, for example, an operation by voice, an operation by eye contact, or an operation by pointing a finger. Furthermore, the type of icon displayed on the display screen may be changed depending on the type of surrounding object or the type of information presented when the user performs an operation on the icon.

[0053] Here, the surrounding object guidance is not limited to a binary choice of "on" or "off." When providing information about surrounding objects, for example, two modes may be provided: one in which surrounding object information is displayed from the beginning, and one in which an icon is displayed first. Depending on the user's selection or a switch based on predetermined conditions, either a process of displaying surrounding object information from the beginning or a process of displaying only an icon indicating that there is surrounding object information that can be displayed may be performed.

[0054] Furthermore, when providing guidance about surrounding objects, three modes may be prepared, for example, a mode in which surrounding object information is displayed from the beginning, a mode in which an icon is displayed first, and a mode in which no icon is displayed at all, and processing may be performed in one of the modes. Furthermore, even in a mode where even an icon is not displayed, if a user inquires, for example, "What is the reputation of this coffee shop?", information about the coffee shop may be presented.

[0055] Furthermore, even in a case where an icon is not even displayed, if the user performs a predetermined action (such as directing their gaze for more than a predetermined period of time) on a surrounding object to which surrounding object information is associated, the surrounding object information may be presented. In other words, even in a case where an icon is not even displayed, if it is detected that the user is interested in a surrounding object to which surrounding object information is associated, the surrounding object information may be displayed.

[0056] 9 to 11 are diagrams showing other display examples on the display screen of the terminal device 200. In FIG. FIG. 9 shows an example of a display on the terminal device 200 of a user who has a meal planned (which may be a plan that has already been completed or may be a plan that is yet to be completed) within a predetermined time from now. More specifically, FIG. 9 shows an example of a display on the terminal device 200 of a user who has a reservation at a restaurant or other eating establishment within a predetermined time from now.

[0057] The display example of FIG. 9 may be displayed on the terminal device 200 of a user who can be confirmed to have actually visited the restaurant within a predetermined time from the present. In this case, information about eating and drinking establishments such as restaurants will not be displayed on the display screen of the terminal device 200 of this user, and in FIG. 9, surrounding object information corresponding to eating and drinking establishments such as restaurants will not be displayed. More specifically, in this example, even though a restaurant exists at the location indicated by reference numeral 9A, information about this restaurant is not displayed.

[0058] FIG. 10 shows a display screen of the terminal device 200 owned by a user who does not have much time to spare and is traveling on foot. For a user who is short on time, guidance is not provided for nearby objects that are visible to the user but require crossing the road to reach. Specifically, although there is a hotel at the location indicated by reference numeral 10A, no information about this hotel is provided. Furthermore, in this embodiment, in the case of a user who does not have much time to spare, guidance is not provided for surrounding objects that are outside the user's travel route even if they are visible to the user.

[0059] FIG. 11 shows a display screen of a terminal device 200 owned by a user who is traveling. Specifically, FIG. 11 shows a display screen of the terminal device 200 owned by a user whose schedule information includes information about a trip to London. In this display example, hotel information is provided as indicated by reference numeral 11A. More specifically, this display example illustrates a case where, when the user passes by a hotel with better conditions than the hotel (not shown) where the user has made a reservation and is planning to stay, information about the hotel with better conditions is provided.

[0060] In this embodiment, the CPU 111 determines the surrounding object information to be presented to the user based on the user's schedule information. In the example shown in FIG. 11, the plan specified by the user's plan information is a trip, so the surrounding object information to be presented to the user is determined based on this information about the trip. In this way, in this embodiment, hotel information is presented to the user.

[0061] More specifically, this example illustrates a case where the user is presented with information about hotels that are cheaper than the hotel that the user has already booked or that are located closer to the hotel that the user plans to stay at. Here, the planned hotel for staying is information about an item that is specified in the user's schedule information and is included in the user's schedule information. On the other hand, the above-mentioned hotel with favorable conditions is not clearly indicated in the user's schedule information, and the information about the hotel with favorable conditions is information about surrounding areas that are not included in the user's schedule information.

[0062] In this embodiment, the CPU 111, which is an example of a processor, of the external server 100 acquires schedule information of the user when presenting surrounding object information. Then, the CPU 111 determines information about the surroundings of the user's current location to be presented to the user based on the schedule information of the user.

[0063] Here, the surrounding object information presented to the user is "information about surrounding objects not included in the schedule information," which means, for example, that information about facilities that are not registered in the user's calendar or information about facilities that are not listed in the user's itinerary is presented to the user when the user comes near these facilities. On the other hand, for example, presenting a user with information about a facility (e.g., a hotel) based on reservation information for the facility when the user is near the facility (e.g., the name of the hotel, its location, and its direction) does not mean presenting the user with information about surrounding objects that are not included in the schedule information based on the user's schedule information.

[0064] FIG. 12 is a diagram showing another example of a display on the user's terminal device 200. In FIG. This display example shows a display screen on the terminal device 200 of a user who frequently uses the store indicated by the reference numeral 12A in the drawing. This display screen does not display basic store information, such as the store name and store ratings.

[0065] On the other hand, this display screen displays information related to the timing at which the user passes through the store. Specifically, in this display example, at the time the user passes through the store, the price of the drink is 10% off, and information indicating that the price of the drink is 10% off is displayed.

[0066] FIG. 13 is a diagram showing another example of a display on the user's terminal device 200. In FIG. In this example display, information about a hotel is displayed. Specifically, the user whose terminal device 200 displays this display screen is currently traveling, and the user's schedule information does not include information about specific accommodation facilities. In this case, in this embodiment, as indicated by reference numeral 13A, the availability of hotels in the vicinity of the user today and the rates for using these hotels today are displayed.

[0067] In this embodiment, peripheral object information is presented to the user in this manner. The presentation of the surrounding object information may be free of charge or may be subject to a fee. Furthermore, if the presentation of the surrounding object information requires a fee, the fee may be charged to a person who wishes to receive the surrounding object information, or to a person who wishes to receive the surrounding object information.Furthermore, the fee may be charged to both the person who wishes to receive the surrounding object information and the person who wishes to receive the information.

[0068] In addition, when charging a delivery requester, the charge may be based on the number of times the nearby object information is delivered. Here, the distribution applicant wants to distribute the surrounding object information to users who are likely to respond to the distributed surrounding object information. Furthermore, the recipient does not want to receive surrounding object information that he or she does not want or receive at an inconvenient time, but rather wants to receive the surrounding object information that he or she wants at the right time.

[0069] Examples of information that can be presented to users as nearby information include the facility name, business days, business hours, congestion status, holidays, services offered (menus, products for sale, etc.), recommended services, coupons, reviews, interior photos, exterior photos, location on a map, address, telephone number, official website URL, social media URL, and social media posts. The information presented to the user as surrounding object information may be information registered by the facility operator in the information processing system 1 of this embodiment, or information registered by the facility user in the information processing system 1 of this embodiment, or information automatically collected from users who have agreed to the terms of use for personal information, or information generated based on this information.

[0070] For example, information regarding the congestion status of a facility may be information registered in a database by the facility operator, information registered in a database by facility users, or information automatically determined based on the number of people staying at the facility, the number of terminal devices 200 owned by people staying at the facility, the number of orders placed online, etc.

[0071] In addition, a promotion function may be provided for businesses that wish to receive information preferentially over other businesses. In this case, information of the event participant who desires priority information distribution is presented in preference to information of other businesses. In this case, a fee may be charged to those who wish to receive information preferentially.

[0072] Possible means of presenting information about surrounding objects to users include information distribution services via email, chat, notification functions on wearable terminal devices or smartphones, and applications that are highly compatible with information distribution. Examples of applications that are highly compatible with information distribution include applications that have search functions, map functions, and route guidance functions. In this embodiment, a case will be exemplified in which information about surrounding objects and information for route guidance is distributed on a navigation application having a route guidance function.

[0073] [Description of Terminal Device 200] FIG. 14 is a diagram showing the hardware configuration of the terminal device 200. As shown in FIG. The terminal device 200 is an example of an information processing device. The terminal device 200 may be a smartphone type, a pair of glasses type, a wrist-worn type, or any other type, and may be a device that can be carried by a user. The terminal device 200 has hardware such as a CPU 211A as an example of a processor, a ROM 211B, a RAM 211C, an information storage device 212, a sensor 213, a camera 214 (an example of a visual information input means), a microphone 215 (an example of an audio information input means), a speaker 216 (an example of an audio information output means), a display 217 (an example of a visual information output means), and a projector 218 (an example of a visual information output means). Note that the terminal device 200 does not need to have all of these. Examples of the sensor 213 include a GPS sensor, a direction sensor, a sensor for acquiring vital information, etc. Examples of the display 217 include a display that allows a view behind the display to be seen through, and a non-see-through display such as a liquid crystal display.

[0074] Here, the program executed by CPU211A can be provided to terminal device 200 in a state where it is stored on a computer-readable recording medium such as a magnetic recording medium (such as a magnetic tape or a magnetic disk), an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory. Furthermore, the program executed by the CPU 211A may be provided to the terminal device 200 using a communication means such as the Internet.

[0075] In this specification, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). Furthermore, the operations of the processors may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. The order of the operations of the processors is not limited to the order described in this embodiment, and may be changed.

[0076] Various applications such as email and calendar applications are pre-installed on the terminal device 200, or may be additionally installed. The terminal device 200 also has a communication interface (not shown) that conforms to communication standards (Wi-Fi, 3G, 4G, 5G, Bluetooth (registered trademark), NFC, etc.), and connects to external devices and networks to send and receive data. It should be noted that the terminal device 200 does not need to have all of the above configurations, and some functions may be omitted depending on the budget and the user's wishes.

[0077] The terminal device 200, such as the eyeglass-type terminal device 200A and the wrist-worn type terminal device 200B shown in FIG. 1, is wearable and is worn on a part of the user's body. The wearable terminal device 200 is not limited to glasses or wristwear, and may also take the form of shoes, clothes, a ring, a headset, contact lenses, or the like.

[0078] The terminal device 200 can visually present information to the user by displaying the information on the display 217. If the terminal device 200 is a glasses-type or contact lens-type device, a display that the user can see through is installed as the display 217 in the terminal device 200, and the user can see what is behind the display through this see-through display. In other words, the terminal device 200 may have a transparent part that the user can see through, and the user can see the real space through this transparent part. When such a see-through display is installed as the display 217 of the terminal device 200, when information is displayed on the display 217, the user can visually recognize both the information displayed on the display 217 and the real space. In this case, information about the surrounding objects around the user can be associated with real objects in the real space that the user visually recognizes via the display 217.

[0079] In this embodiment, the CPU 111 associates peripheral object information with real objects in real space that the user visually recognizes through the see-through display 217. However, the display 217 may not be see-through, such as a liquid crystal display. For example, the display of the wrist-worn terminal device 200B in FIG. 1 and the display of a smartphone are usually not see-through. In the terminal device 200 having a non-see-through display, it is possible to associate peripheral objects around the user with information about the peripheral objects on an image obtained by the camera 214 that captures the user's surroundings.

[0080] Here, the eyeglass-type or contact lens-type terminal device 200 can be said to be a device worn in front of the user's eyeballs. In the glasses-type or contact lens-type terminal device 200, the see-through display 217 is placed in front of the user's eyeball, and this terminal device 200 can be said to be a device in which at least a portion is placed in front of the user's eyeball.

[0081] In this embodiment, the camera 214 of the terminal device 200 captures the situation within the user's field of view. In this embodiment, image data acquired by the camera 214 provided in the terminal device 200 is analyzed, for example, by an image analysis function of the terminal device 200. More specifically, the image data acquired by the camera 214 of the terminal device 200 is analyzed by the CPU 211A of the terminal device 200. However, the image data acquired by the camera 214 of the terminal device 200 may be analyzed by another terminal device 200 different from this terminal device 200 or by the CPU 111 provided in the external server 100 .

[0082] In this embodiment, the analysis identifies the user's surroundings. Specifically, in this embodiment, for example, a known image analysis technique is used to recognize facilities included in the image data as objects in the user's vicinity. In other words, in this embodiment, recognition of facilities within the user's field of view is performed. Alternatively, the user's surroundings may be identified based on the user's current location, the direction the user is facing, and an electronic map. Specifically, in this case, an electronic map is referenced, and facilities within the user's field of view are identified based on the user's current location and the direction in which the user is facing.

[0083] In this embodiment, image data acquired by the camera 214 of the terminal device 200 is stored as one or more video files in the information storage device 212 of the terminal device 200 or in the information storage device of another device such as the external server 100. However, the image data acquired by the camera 214 of the terminal device 200 may be saved as a still image file.

[0084] Here, an example of the sensor 213 provided in the terminal device 200 is the sensor 213 that acquires vital information of the user, as described above. Sensors 213 that acquire vital information include sensors that detect the rotation and direction of the user's head, sensors that detect the direction of the user's gaze, sensors that work in conjunction with GPS to identify the user's location on Earth, and sensors that measure the wearer's vital signs (heart rate, blood pressure, breathing, body temperature, blood sugar level, oxygen level, state of consciousness, brain waves, pupil response, etc.).

[0085] Additionally, the sensor 213 that acquires the user's vital information and the like is not limited to being installed in the terminal device 200 held by the user, but may be installed in a location away from the user. For example, a sensor 213 for detecting the body temperature of a user may be provided in a device installed at the entrance of a store, and this sensor 213 may measure the body temperature of the user. Furthermore, for example, a sensor 213 for acquiring information such as the user's physique may be installed at a location different from the user, and this sensor 213 may acquire information about the user's physique.

[0086] Here, as described above, a contact lens type device can also be cited as an example of the terminal device 200 carried by the user. This contact lens type device can also be said to be a device at least a part of which is placed in front of the user's eyeball. In the case of this contact lens type device, this device mainly functions as a display device, but the display control in the contact lens type device may be performed by a device other than this device. Examples of such other devices include a smartphone, a user's PC, and an external server 100.

[0087] Furthermore, the various functions provided in this embodiment may be provided as functions of the OS of the external server 100, or may be provided as applications that run on the OS. Furthermore, the various functions provided in this embodiment may be provided independently by one or more terminal devices 200, or may be provided by two or more terminal devices 200 working together. In this case, the various functions may be provided as functions of the OS of the terminal device 200, or may be provided by an application running on the OS. Furthermore, in this embodiment, various functions are provided by the navigation application installed in the terminal device 200, but the functions of this embodiment may be provided by other applications.

[0088] The navigation application installed in the terminal device 200 of this embodiment has two functions. Specifically, this navigation application has a route guidance function that provides guidance on the route to a destination, and a surrounding object information presentation function that presents surrounding object information, which is information about surrounding objects that exist around the user.

[0089] The route guidance function is a function that presents the user with the route and direction that the user should take. In this embodiment, the route guidance function displays a message for guiding the user along the route, as shown in FIG. 3 and the like. The surrounding object information presentation function is a function that presents to the user surrounding object information, which is information about surrounding objects that exist around the user's current location. Note that the surrounding object information presentation function may be linked to an external information distribution system that is provided separately from the information processing system 1 of this embodiment.

[0090] Each of these two functions, the route guidance function and the surrounding object information presentation function, may be configured to be switchable between enabled and disabled in response to an instruction from the user. Furthermore, each of these two functions may be enabled or disabled depending on whether a predetermined condition other than a user instruction is satisfied. In this embodiment, it is possible to enable both the route guidance function and the surrounding object information presentation function, enable only one of them, or disable both of them.

[0091] In addition, if the function is configured to be switched between valid and invalid depending on whether or not a condition is met, the function may be switched between valid and invalid when the condition is met and the user's consent is also obtained. Specifically, in this case, when the conditions are met, the system will ask the user questions such as, "Would you like me to give you directions?" or "Would you like to disable the surrounding object information presentation function?", and if the user responds to this question, the system will switch between enabled and disabled.

[0092] Examples of surrounding objects include man-made objects such as facilities and natural objects such as plants and trees. Examples of facilities include restaurants, hotels, supermarkets, museums, schools, and observation decks. Facilities provide various services. For example, a restaurant is a service provider that provides food and drinks, and a supermarket is a service provider that sells food and daily necessities.

[0093] The surrounding object information presented by the surrounding object information presentation function includes, for example, the facility name (e.g., Tokyo Tower), type (e.g., restaurant, hotel, park, school, tourist attraction, etc.), and rating (e.g., score or free-form comment). Note that the information presented by the surrounding object information presentation function is not limited to these. If the surrounding object is a facility registered in an external information distribution system, information provided by this external information distribution system may be presented as surrounding object information.

[0094] As shown in FIG. 1, an example of the terminal device 200 is a glasses-type terminal device 200A that is worn on the head of a user. In this embodiment, when a user wears such a glasses-type terminal device 200 and walks around town, information about surrounding objects within the user's field of view is displayed on the display 217 of the glasses-type terminal device 200. More specifically, surrounding object information is displayed in association with surrounding objects within the user's field of view. Note that the surrounding object information may be presented on another terminal device 200 carried by the user (for example, a smartphone-type terminal device 200 or a wrist-worn terminal device 200) in addition to or instead of the glasses-type terminal device 200.

[0095] FIG. 15 is a flowchart showing an example of the flow of processing executed in this embodiment. In the flowchart shown in FIG. 15, it is assumed that the user is wearing the terminal device 200 in the form of glasses. In this processing example, when it is detected that a user wearing the glasses-type terminal device 200 has started to move (step S101), the destination of the user is identified based on the user's schedule information (step S102).

[0096] Next, it is determined whether route guidance should be provided to the user (step S103), and if it is determined that route guidance should be provided, the route guidance function is enabled (step S104), after which the process proceeds to step S105. On the other hand, if it is determined in step S103 that route guidance should not be provided, the route guidance function is not enabled and the process proceeds to step S105.

[0097] In step S105, it is determined whether or not surrounding object information should be presented to the user (step S105). If it is determined that the surrounding object information should be presented, the surrounding object information presentation function is enabled (step S106), and the process proceeds to step S107. On the other hand, if it is not determined in step S105 that the surrounding object information should be presented, the surrounding object information presentation function is not enabled and the process proceeds to step S107. Thereafter, in this embodiment, when the end of the user's movement is detected (step S107), the route guidance function and the surrounding object information presentation function are disabled (step S108). More specifically, when the end of the user's movement is detected, if the route guidance function and the surrounding object information presentation function are enabled, these functions are disabled.

[0098] FIG. 16 is a flowchart showing another processing example. 16, first, the user's past movement history is acquired (step S201). Specifically, for example, past information obtained by a GPS provided in the eyeglass-type terminal device 200 and stored in the information storage device 102 of the external server 100 is acquired, and information about the user's past movement history is acquired. In this embodiment, information about the user's past movement history is stored in the information storage device 102 of the external server 100, and information about the user's past movement history can be obtained by referring to the information stored in this information storage device 102. Next, information obtained by a GPS provided in the eyeglass-type terminal device 200 is acquired to identify the current location of the user (step S202).

[0099] In step S201, other actual information may be acquired instead of or in addition to acquiring the movement history. Furthermore, the order of processing is not limited to the above, and the user's current location may be identified first, and then information about the user's past movement history may be obtained.

[0100] In step S203, it is determined whether the user has traveled the route from the current location to the user's destination with a frequency equal to or greater than a predetermined threshold (step S203). Then, if it is determined in step S203 that the user has traveled the route from the current location to the destination with a frequency equal to or greater than a predetermined threshold, it is determined not to provide route guidance to the user (step S204). On the other hand, if it is determined in step S203 that the user has not traveled the route from the current location to the destination with a frequency equal to or greater than the predetermined threshold, it is determined to provide route guidance to the user (step S205). In this case, the route guidance function is enabled.

[0101] FIG. 17 is a flowchart showing another processing example. In this processing example, first, based on the user's schedule information, information about the location of the user's next schedule and information about the start time of this next schedule are acquired (step S301). Note that information about the end time may be acquired in addition to the start time. Also, information about both the start time and the end time may be acquired. Next, information about the current time and information about the user's current location are obtained (step S302).

[0102] In other words, in this embodiment, in step S301, the CPU 111 of the external server 100 acquires, as the schedule information, at least one of the start time and end time of the schedule and information about the location where the schedule is to be held. The order of processing is not limited to this, and information about the current time and current location may be obtained first, and then information about the location of the next appointment and the start time of the next appointment may be obtained.

[0103] Next, in this processing example, in step S303, it is determined whether there is sufficient time remaining. The "remaining time" refers to the time obtained by subtracting the time required to travel from the current location to the location of the next scheduled event from the remaining time from the current time until the start time of the next scheduled event. In this embodiment, if it is determined in step S303 that there is sufficient time remaining, the CPU 111 determines to present nearby object information (step S304). In other words, in this case, the CPU 111 determines to present information about the user's surroundings to the user, and the surroundings information presentation function is enabled.

[0104] On the other hand, if it is not determined in step S303 that there is sufficient time remaining, the CPU 111 determines not to present the surrounding object information (step S305).

[0105] In this embodiment, the CPU 111 determines the surrounding object information to be presented to the user based on the remaining time from the current time until the scheduled start time and the time required to travel from the current location to the scheduled event location. In addition, CPU 111 may control the content displayed on the display screen so that the amount of information displayed on the display screen decreases as the proportion of the time required to travel from the current location to the event location increases relative to the remaining time from the current time to the start time.

[0106] Here, if processing is performed so that the amount of information on the display screen decreases as the proportion of the required time to the remaining time increases, for example, when a user travels to a destination, even if a lot of information is displayed immediately after the user starts traveling (for example, even if the control is in the ``more information'' mode), if the user gets stuck at many traffic lights or makes detours during the trip, the proportion of the required time to the remaining time will increase, and the amount of information on the display screen will decrease (for example, by controlling in the ``less information'' mode). Also, for example, if a friend contacts the user along the way and the scheduled time is delayed, or if the user moves at a speed greater than the user's speed that was used as the basis for calculating the required time (for example, if the travel speed was calculated based on walking but the user actually ran), even if less information is displayed immediately after the start of the movement (for example, even if the display is controlled in the "less information" mode), the proportion of the required time to the remaining time will decrease during the movement, and the amount of information on the display screen will increase (for example, by controlling in the "more information" mode).

[0107] Furthermore, in this embodiment, the CPU 111 repeatedly determines the surrounding object information as time passes, even if the user is staying in substantially the same place. In other words, in this embodiment, the CPU 111 makes a decision on presenting surrounding object information every time a predetermined time period elapses, even if the user is staying in substantially the same place and is not moving. As a result, the display on the user's terminal device 200 becomes the latest every time a predetermined period of time elapses.

[0108] FIG. 18 is a flowchart showing another processing example. In this processing example, first, schedule information of the user is acquired (step S401). Next, surrounding objects existing around the user are identified (step S402). Note that the surrounding objects may be identified first, and then the schedule information of the user may be acquired. In the process of step S401, actual information may be acquired instead of or in addition to the user's schedule information.

[0109] Next, in step S403, this process determines whether the user is likely to react to the surrounding object information, which is information about the surrounding object identified in step S402, when the surrounding object information is presented to the user. Here, reacting to the presented information means clicking on the presented link, making a reservation for the presented facility, or visiting the presented facility. If it is determined in step S403 that the user is likely to react, it is determined that the surrounding object information should be presented to the user (step S404). In this case, the surrounding object information is displayed. On the other hand, if it is not determined in step S403 that a reaction is likely, it is determined not to present the surrounding object information to the user (step S405). In this case, the surrounding object information is not displayed. As an example, whether or not a user is likely to react to the presented information can be determined by calculating the likelihood that the user will react to the presented information and determining whether the value is above or below a threshold.

[0110] FIG. 19 is a flowchart showing another processing example. In this processing example, first, peripheral objects existing around the user are identified (step S501). If there are multiple peripheral objects existing around the user, multiple peripheral objects may be identified. Next, information about the identified peripheral objects (peripheral object information) is acquired. If multiple peripheral objects are identified, information about each of the peripheral objects is acquired (step S502).

[0111] Next, it is determined whether the user is interested in each piece of acquired surrounding object information (step S503). Then, for surrounding object information determined to be of interest to the user, it is determined to present the surrounding object information to the user (step S504). In this case, this surrounding object information is presented to the user. On the other hand, it is determined that the surrounding object information that is not determined to be of interest to the user is not to be presented to the user (step S505). In this case, this surrounding object information is not presented to the user.

[0112] In this embodiment, the CPU 111 acquires the user's interest level information for each surrounding object. Then, the CPU 111 determines the display content on the display screen based on the schedule information and the interest level information.

[0113] It should be noted that the content to be displayed on the display screen may be determined based only on the interest level information, rather than based on both the schedule information and the interest level information. Furthermore, when identifying what the user is interested in, a specific facility (e.g., Restaurant N) or a specific product (e.g., lunch course A at Restaurant N) may be identified, or a larger object of interest to the user (e.g., gourmet food) may be identified. In the processing example shown in FIG. 19, information about surrounding objects in which the user is interested is presented to the user. In the processing examples of Figures 17 to 19, a decision is made regarding the presentation of information about surrounding objects identified as being in the user's vicinity. However, as will be described later, when making a decision regarding the presentation of information about related objects that are not in the user's vicinity but have a predetermined relationship with the surrounding objects in addition to, or instead of, the information about the surrounding objects, a decision regarding the presentation of information about related objects can be made based on planned information, actual information, or both, with reference to the processing of Figures 17 to 19.

[0114] A specific example of the process will be described with reference to FIGS. 20 and 21 are diagrams showing the user's travel route. First, the premise of the process will be explained. The following describes the process when user Y moves around town. According to User Y's calendar (an example of schedule information), User Y's schedule for Sunday, March 14th includes a scheduled lunch with Friend S at Restaurant N (see FIG. 20) from 12:00 to 14:30. In other words, according to the schedule information of user Y, user Y is scheduled to have dinner with friend S at a restaurant called Restaurant N from 12:00 to 14:30 on Sunday, March 14th. Restaurant N is a five-minute walk from Shibuya Station. User Y lives an eight-minute walk from Ikebukuro Station (Figure 21), which is a ten-minute train ride from Shibuya Station.

[0115] User Y's actions on Sunday, March 14th are as follows: 10:30 Leave home 10:40 Take the train from Ikebukuro Station, the nearest station, to Shibuya Station. 10:50 Arrive at Shibuya Station (see Figure 20). Head to clothing store T. After getting off the train at Shibuya Station, User Y starts walking towards Clothing Store T, which he has been wanting to visit for some time, even though it is not in his plans. As shown in Figure 20, Clothing Store T is located on the opposite side of Shibuya Station from Restaurant N. ·10:55 Arrive at clothing store T. User Y purchases a suit from clothing store T and requests that it be hemmed. The hemming was scheduled to be completed by 3:00 PM on Sunday, March 14th. User Y received an email containing the store name and the collection period. The end of the exchange period (an example of a deadline) was one week later.

[0116] 11:20 Depart from clothing store T and start walking in the direction of restaurant N. The shortest travel time from clothing store T to restaurant N is 15 minutes. 11:35 Arrived at Park S. Kill time. ·11:55 Arrived at Restaurant N. 14:30 Depart from Restaurant N. ·14:35 Stop by Bookstore E. 15:40 Depart from Bookstore E. 15:00 Arrive at the clothing store T and pick up your suit after it has been hemmed. 15:05 Depart from clothing store T. 15:10 Arrive at Shibuya Station. 15:15 Take the train from Shibuya Station to Ikebukuro Station. 15:25 Arrived at Ikebukuro Station (see Figure 21). Started walking. 15:30 Arrive at Bakery K and buy bread. 15:45 Arrive at Supermarket F. Buy ingredients for dinner. ·16:15 Arrive home.

[0117] Based on the above assumptions, the processing executed in this embodiment will be described in detail. [Check using a route search application on your smartphone] Before leaving his home (see Figure 21), User Y searched for a route from his home to Restaurant N using a route search app on his smartphone. Then, from among the multiple search results, User Y selected the route "Sunday, March 14th, walk (8 minutes) → JR Yamanote Line (10:40 Ikebukuro Station → 10:50 Shibuya Station) → walk (5 minutes)."

[0118] Furthermore, User Y carried the smartphone without closing the route search application, leaving the display screen showing the selected route open. In other words, User Y carried the smartphone with the route search application active. "JR Yamanote Line (10:40 Ikebukuro Station → 10:50 Shibuya Station)" is not schedule information obtained from user Y's calendar, but is schedule information obtained from user Y's selection history, and is an example of user Y's schedule information.

[0119] [Wearing a wearable device] On this day, user Y carried a smartphone-type terminal device 200 as the terminal device 200, and also wore a glasses-type terminal device 200A and a wrist-worn terminal device 200B connected to the smartphone-type terminal device 200 via wireless communication (e.g., Bluetooth), and spent the day that day. In this processing example, the camera 214 of the eyeglass-type terminal device 200 (see FIG. 1) acquires information about what is in the field of view of the user Y. In addition, in this processing example, the camera 214 and sensor 213 of the eyeglass-type terminal device 200 acquire user Y's location information, user Y's vital signs (heart rate, blood pressure, breathing, body temperature, blood sugar level, oxygen level, state of consciousness, brain waves, pupil response, etc.), etc. The acquired information is stored in, for example, an information storage device 102 provided in the external server 100.

[0120] In this processing example, the sensor of the wrist-worn terminal device 200 also acquires the user Y's position information and vital signs (heart rate, blood pressure, respiration, body temperature, blood sugar level, oxygen level, state of consciousness, etc.). The acquired information is similarly stored in, for example, the information storage device 102 of the external server 100 .

[0121] The information storage device 102 also stores information previously acquired about the user Y. In this embodiment, the status of user Y and the things that user Y is interested in can be identified based on the information stored in the information storage device 102. In other words, in this embodiment, information indicating interest in an object at a past time when the object was in the user's field of view is stored in the information storage device 102 .

[0122] In this embodiment, the content to be displayed on the display screen is determined taking into consideration interest level information, which indicates the user's interest in an object at a time when the object was in the user's field of view. In addition, when identifying the status of user Y or the subject of user Y's interest, the information stored in the information storage device 102 may be used alone, or multiple pieces of information stored in the information storage device 102 may be used in combination.

[0123] In this embodiment, for example, if user Y looks at a specific surrounding object for more than a predetermined time, information about this specific surrounding object and the vital signs obtained while looking at this specific surrounding object are stored in the information storage device 102 in an associated state. In this case, by obtaining this information stored in the information storage device 102, it is possible to identify what User Y is interested in. In other words, in this case, the CPU 111 uses vital information obtained from the user's living body, and identifies what the user is interested in based on this vital information.

[0124] Specifically, in this case, for example, if the heart rate included in the vital signs associated with a particular surrounding object is higher than the normal average value, it can be determined that user Y is interested in this particular surrounding object. Furthermore, when the vital signs of user Y are acquired as in this embodiment, the degree of fatigue, hunger, whether or not user Y has a fever, etc. can be determined based on the vital signs.

[0125] [Location tracking] User Y enabled the location information function (for example, GPS) in at least one of the terminal devices 200 that travel with the user. User Y also spent the day with the location information function enabled. The location information of the user Y is stored in an information storage device 102 provided in the external server 100 . In addition, the information storage device 102 also stores past location information obtained for the user Y.

[0126] In this embodiment, based on this past location information, the past behavioral trends of user Y are analyzed, the future behavior of user Y is predicted, and information about the uniqueness of user Y's current behavior is obtained. The specificity of user Y's current behavior is determined based on, for example, information about whether the route user Y is on is the same as the route user Y usually takes, or information about whether user Y's current location is a place user Y usually visits.

[0127] If the road where user Y is on is the same as the road that user Y usually takes, or if user Y's current location is a place that user Y usually visits, the specificity will be small. Furthermore, if the road on which user Y is currently located is not a road that user Y normally takes, or if user Y's current location is not a place that user Y normally visits, the specificity will be large.

[0128] In addition, in this embodiment, by referring to the user's past location information, it is possible to identify schedules that are not registered in the user's calendar (an example of schedule information) but are actually included as part of the user's schedule. Specifically, for example, if information such as being in a specific location at a specific time on a specific day is obtained based on past location information, it is possible to identify the user's actual plans to be in a specific location at a specific time on a specific day.

[0129] User Y's past location information can also be said to be information about User Y's behavior history. If the location specified by the location information of user Y is, for example, a restaurant, it can be estimated that user Y ate a meal at this restaurant. Additionally, the behavioral history of user Y can also be identified based on the purchase history at the store, the payment history when using an electronic payment system, the reservation history in a reservation system, and so on.

[0130] [Leaving home and heading to the nearest station, Ikebukuro Station] User Y frequently travels on a route between his home (see FIG. 21) and Ikebukuro Station, the nearest station. Specifically, User Y often takes the train from Ikebukuro Station when commuting to work or when going to other stations, and frequently travels the route connecting his home and the nearest station, Ikebukuro Station (the road between his home and Ikebukuro Station).

[0131] Here, if the user is on a route that the user frequently takes or if the user is within the user's daily range of activity, processing may be performed such that surrounding object information is not presented to the user. In addition, if the user is on a route that the user frequently takes or is within the user's range of daily activities, processing may be performed to present only specific types of surrounding object information to the user. Specifically, for example, a process may be performed in which only information (such as campaign information) that is valid only within a predetermined period of time that includes the timing when the user passes through the surrounding object is presented.

[0132] For example, even if user Y is presented with information such as "Business hours: 9:00-20:00 (open all year round)" and "Rating: ★★★★☆" for a store on a route that user Y passes through almost every day, this information is likely not useful to user Y. Information useful to User Y includes, for example, information about something unusual, special, or irregular at the store. For example, information such as "Today only! Eggs for 98 yen per pack," "Happy hour is currently on," or "We will be closed on March 31st for fiscal year-end settlement." Information about ongoing special sales can also be said to be valid only for a predetermined period of time, including the time when User Y passes by the store.

[0133] In view of the above circumstances, when presenting surrounding object information, it is preferable to change the content of the surrounding object information to be presented depending on the user information, rather than presenting the surrounding object information uniformly. In other words, when presenting surrounding object information, it is preferable that specific surrounding object information among the surrounding object information is displayed in preference to other surrounding object information, depending on the user's information.

[0134] In addition, the user may be asked to input the routes that the user frequently takes and the range of the user's daily activities, and based on this input information, the routes that the user frequently takes and the range of the user's daily activities may be identified. When the user is to input information about a route that the user frequently takes or the range of daily activities of the user, the user may be asked to specify the route or the range of activities, for example.

[0135] Specifically, for example, the user may be asked to set the location of the nearest station or the location of his / her home, and the route connecting the nearest station and the location of his / her home may be set as the route that is frequently taken. Furthermore, a certain range including a route connecting the nearest station and the user's home may be set as the user's daily range of activity. Furthermore, the routes that the user frequently takes and the user's daily range of activities may be identified based on the user's past movement history and behavior history.

[0136] Furthermore, when presenting the surrounding object information to the user, the user may determine whether the user is "going out or coming back." In other words, when presenting the surrounding object information to the user, it may be determined whether the user is on the way to a destination or on the way back home.

[0137] For example, if user Y leaves his / her home and is on his / her way to the nearest Ikebukuro station, it can be considered that the user is on his / her way to his / her destination, and in this case, he / she is not on his / her way home. In this case, even if user Y is planning to buy fresh ingredients for that day's dinner at a supermarket along the route, it is unlikely that the user would buy fresh food on the way to the destination. For this reason, it is preferable to determine whether the user is heading out or coming back, and to decide whether to present the surrounding object information based on the result of this determination. Preferably, information about the fresh ingredients for dinner is presented to the user on the way home.

[0138] Whether the user is on the way to a destination or on the way home is determined based on the user's direction of movement, behavioral trends determined from the user's movement history and behavior history, user schedule information, etc. These pieces of information may be used alone or in combination. When it is based on the user's schedule information, for example, it is determined whether the user has any plans after the current time. For example, if the user has an appointment after the current time, it is determined that the user is on the way to a destination. Also, if the user has no appointment after the current time, it is determined that the user is on the way home.

[0139] In other words, the user may determine based on the schedule information whether or not they will pass through a route that includes the target supermarket within a predetermined period of time, and the content of the surrounding object information may be changed depending on the result of this determination. Specifically, for example, if the user is in front of a supermarket and has no plans for the user after the current time, it is determined that the user will not pass through a route that includes the supermarket within a predetermined period of time.In this case, information about the supermarket is presented. More specifically, if the user is in front of a supermarket and has no plans for the day after the current time, it is determined that the user will not pass through a route that includes the supermarket within a predetermined period (for example, until midnight on that day).In this case, information about the supermarket is presented.

[0140] On the other hand, for example, if the user is in front of a supermarket and has plans to go there after the current time, it is determined that the user will again pass through a route that includes the supermarket within a predetermined period of time. More specifically, if the user is in front of a supermarket and has plans for the day after the current time, it is determined that the user will again pass through the route that includes the supermarket within a predetermined period (for example, until midnight on that day). In this case, the information about the supermarket is not presented at this time, but when the user returns to the supermarket later, the information about the supermarket is presented.

[0141] Examples of information that may be presented to a user on their way home include information about heavy items, information about items that need to be refrigerated or frozen, and information about items that are available at any time and have no particular sales hours. Other information that may be preferable to present to the user on their way home includes, for example, information about products that are only on sale during the time the user is expected to return home, information about products that are particularly cheap during the time the user is expected to return home, information about products that the user will buy for themselves, and information about products that the user will buy for their family members.

[0142] Examples of information that may be preferable to present to a user while they are on their way out include information about limited-time sales that will not be running when the user gets home, or information indicating that a store will be closed when the user gets home. Other information that may be preferable to be presented to a user while they are out and about includes, for example, information about items that will be used for that day's plans, information about stores that sell items that will be used for that day's plans, information about products that are not available near the location where the day's plans are being held, and information about stores that sell items that are not available near the location where the day's plans are being held. Examples of items used for the day's schedule include items needed for the day's schedule and gifts to be given to people who are to be met that day.

[0143] The specific processing will be further explained. In this processing example, user Y leaves home at 10:30, and if user Y takes the JR Yamanote Line (10:40 Ikebukuro → 10:50 Shibuya) which is included in the selected route, user Y will have very little time to spare. In this case, user Y does not have time to stop by and shop at stores along the route. In such a situation, the CPU 111 performs processing to prevent the presentation of surrounding object information or to reduce the amount of surrounding object information to be presented.

[0144] In other words, in this embodiment, the difference between the time required to travel from user Y's current location to the location of the next scheduled appointment (here, Ikebukuro Station) and the time from the current time to the start time of the next scheduled appointment (here, the train departure time of 10:40) is obtained, and user Y's leeway level is obtained based on this difference. Then, based on this margin, a decision is made as to whether or not to present surrounding object information, and the content and amount of surrounding object information.

[0145] [Take the train from Ikebukuro Station to Shibuya Station] The train route from Ikebukuro Station to Shibuya Station is one of the common routes for user Y to travel from his / her home to restaurant N. The route from Ikebukuro Station to Shibuya Station is a route heading in the direction of restaurant N. This route from Ikebukuro Station to Shibuya Station is not a walking route, but a route that uses public transportation. The train that runs along this route is a means of transportation that travels faster than walking. Furthermore, the train that runs along this route is a public transportation system, and is a means of transportation that cannot be stopped at the will of an individual.

[0146] Between Ikebukuro Station and Shibuya Station is Shinjuku Station (not shown), and right next to Shinjuku Station is the Shinjuku branch of the department store I. As the train that User Y is riding approaches Shinjuku Station, the "Department Store I" sign hanging on the Shinjuku branch of Department Store I comes into User Y's field of vision.

[0147] In this embodiment, the CPU 111 controls so that the surrounding object information is not displayed when the transportation mode currently used by the user is a specific transportation mode. Note that if the user subsequently starts traveling by a transportation mode other than the specific transportation mode, the CPU 111 controls so that the surrounding object information is displayed again. Specifically, in this embodiment, when it is detected that the user is on a train, the surrounding object information presentation function is disabled. In this embodiment, when the user is on a train, the surrounding object information presentation function is disabled because the user cannot access surrounding objects. In addition, in this embodiment, when it is detected that the user has gotten off the train, the surrounding object information presentation function is enabled.

[0148] In this embodiment, the means of transportation used by the user is identified from the user's current location, travel speed, passing points, electronic payment history information (information such as payment terminal, payment amount, payment location, etc.), time information for the means of transportation, usage status of the route guidance application, search history of the route guidance application, etc. Examples of transportation include walking, private car, taxi, train, plane, ferry, etc., but transportation is not limited to these. Furthermore, transportation means can be classified into various categories such as "ground and underground," "land, water, and air," and "personal transportation and public transportation," but the classification method is not limited to these. Furthermore, the means of transportation may be a specific means of transportation such as "walking" or "JR Yamanote Line," or may be a more general means of transportation such as "any means of transportation that is faster than XX km / h."

[0149] Even if a user on public transportation such as a train is presented with information about the Department Store I Shinjuku store, such as "Current congestion level: low" or "Current parking space: available," the user cannot stop the train wherever they like and therefore cannot stop by the Department Store I Shinjuku immediately. In other words, presenting information about the current situation at the Department Store I Shinjuku store is hardly beneficial to the user. Therefore, it is preferable to control the display so that information about the Department Store I Shinjuku store is not presented even if the Department Store I Shinjuku store comes into the user's field of vision. On the other hand, general information describing the department store I Shinjuku, such as an introduction to the tenants in the department store I Shinjuku and a user's evaluation of the department store I Shinjuku, may be useful information for the user when the user uses the department store I in the future, and may be useful information for the user, so such general information may be provided to the user. In other words, control may be exercised so that only specific types of information about the department store I Shinjuku are presented to a user traveling by a specific means of transportation, and other types of information are not presented.

[0150] Alternatively, for a user who has the Shinjuku branch of the department store I in view, it may be determined whether or not to display specific information based on the schedule information of the user. For example, it may be possible to determine whether or not the user has plans to visit the vicinity of a particular surrounding object within a certain period of time, and to control the display of surrounding object information differently depending on whether or not the user has plans to visit.

[0151] Specifically, suppose that the information about department store I includes limited-time information such as "Coupon: Receive an Eco Bag (registered trademark) with purchases of 3,000 yen or more (available at all department store I stores until March 31st)." If this information is also related to an associated object that has a predetermined relationship with department store I Shinjuku (for example, department store I Shibuya, which is part of the same chain) that is located near or on the route to a place where user Y plans to go, presenting this information to user Y, whose field of vision is within department store I Shinjuku, can encourage the user to stop by department store I Shibuya.

[0152] Additionally, it is preferable that the coupon be presented to a user who has the Shinjuku branch of Department Store I within sight, provided that the date and time the user plans to pass by the Shibuya branch of Department Store I is within the coupon's expiration date.

[0153] [Arriving at Shibuya Station and heading to clothing store T] In this embodiment, after getting off the train at Shibuya Station (see FIG. 20), user Y starts walking toward clothing store T, which is not registered in the calendar as an event but which user Y has wanted to visit for some time. As shown in FIG. 20, clothing store T is located on the opposite side of Shibuya Station from restaurant N. Here, if user Y, who is heading to clothing store T, is presented with information about other stores on the route to clothing store T, or if too much information is presented, user Y may make too many detours to these other stores, and user Y may not be able to achieve his or her original goal of "having lunch at restaurant N."

[0154] Therefore, in this case, when presenting surrounding object information to a user, it is preferable to decide whether to present surrounding object information and how much surrounding object information to present based on (1) the user's current location and (2) the location of the user's next appointment and the start time of this appointment. Specifically, in this case, it is preferable to decide whether or not to present surrounding object information and how much surrounding object information to present based on the difference between the time required to travel from the user's current location to the location of the next appointment (here, restaurant N) and the time remaining from the current time until the start time of the next appointment. Specifically, for example, if the difference is smaller than a threshold value, a decision is made not to present peripheral object information or to reduce the amount of peripheral object information to be presented. Additionally, a process may be performed to reduce the amount of surrounding object information to be presented as the difference becomes smaller.

[0155] Additionally, a decision as to whether or not to present peripheral object information to the user and the extent to which peripheral object information should be presented may be made based on the user's traveling direction. Specifically, for example, if the user is moving off the usual route taken to get to the next scheduled location, or if the user is moving in a direction different from the next scheduled location, a decision is made not to present the user with surrounding object information or to reduce the amount of surrounding object information.

[0156] If the user is traveling off a typical route or in a direction different from the location of the next appointment, it is assumed that the user is traveling to fulfill an appointment that is not registered on the calendar (an example of schedule information) before the next appointment that is registered on the calendar. In this case, presenting information to the user may actually be a nuisance to the user, so as described above, a decision may be made not to present surrounding object information to the user or to reduce the amount of surrounding object information.

[0157] In addition, decisions regarding the presentation of surrounding object information and route guidance information may be made based on the detection results of the sensor 213 installed in the user's terminal device 200 and images obtained by the camera 214 installed in the user's terminal device 200. Specifically, based on information obtained by the sensor 213 and the camera 214, it is possible to detect, for example, that the user is lost, and based on biometric information such as the user's heart rate, it is possible to detect that the user is feeling anxious.

[0158] In this case, for example, processing may be performed such as stopping the presentation of surrounding object information or reducing the amount of surrounding object information presented, or conversely, processing may be performed such that the route guidance function is enabled and route guidance information is displayed. Here, as described above, the route guidance information may include a display indicating the road the user should take, an arrow indicating the direction the user should head, or a warning that the user has deviated from the route.

[0159] Alternatively, for example, information about the user's moving speed may be acquired, and decisions regarding the presentation of surrounding object information and route guidance information may be made based on this information about the moving speed. Specifically, for example, if the user's movement speed is less than a predetermined threshold, it can be assumed that the user is not in a hurry and is traveling while stopping at facilities along the way, and in this case, information about surrounding objects will be presented, and route guidance information will not be displayed.

[0160] Also, for example, if the user's movement speed is equal to or greater than a predetermined threshold, it can be assumed that the user is moving toward a destination, and in this case, for example, information about surrounding objects is not presented, and route guidance information is displayed. In addition to the situation where the user's movement speed decreases when the user is moving and stopping at facilities along the way, the user may also be lost. Therefore, instead of or in addition to information about the movement speed, a decision regarding the presentation of surrounding object information and route guidance information may be made based on images from the sensor 213 or camera 214 provided in the terminal device 200.

[0161] Specifically, if the user is lost, it is conceivable that the user will become anxious and restless, and will frequently look around, or look at signs or maps. If the camera 214 is provided to capture what the user is looking at, it is possible to determine whether the user is frequently looking around, based on whether what is captured in the image of the camera 214 changes frequently in the horizontal direction faster than a threshold, and it is possible to determine whether the user has looked at signs or maps based on how long the signs or maps appear in the image acquired by the camera 214. Furthermore, since the user's anxiety is reflected in the user's vital information (e.g., heart rate), for example, if the heart rate acquired by the sensor 213 provided in the terminal device 200 increases, it is determined that the user is lost. If it is determined that the user is lost, the display of the surrounding object information may be stopped or the amount of the surrounding object information displayed may be reduced. Alternatively, if it is determined that the user is lost, route guidance information may be displayed.

[0162] Conversely, if the camera 214 and sensor 213 provided in the terminal device 200 do not detect any of the above events that indicate that the user is lost, it is determined that the user is not lost. In this case, the surrounding object information is presented or the amount of surrounding object information is increased, or the route guidance information is not displayed or the amount of route guidance information is reduced.

[0163] The CPU 111 of this embodiment identifies the user's condition based on the user's schedule information and at least one of vital information obtained from the user's living body and an image obtained by a camera 214 moving with the user. Then, depending on the identified state, the CPU 111 stops presenting the surrounding object information or reduces the amount of the surrounding object information presented, as described above. In this case, the CPU 111 also causes route guidance information to be presented.

[0164] In other words, the CPU 111 determines that the user is in a predetermined state, such as being lost, based on at least the schedule information. Then, when it is determined that the user is in a predetermined state, the CPU 111 performs control so that, for example, route guidance information is presented to the user.

[0165] [Arrived at clothing store T] User Y purchases a suit from clothing store T and requests hemming. The hemming is completed at 3:00 PM on Sunday, March 14th. User Y also receives an email containing the store name and the suit pickup period. The pickup period is one week. The information in the email received by user Y, "Pickup period: one week from 15:00 on Sunday, March 14th," does not fall under "schedule information obtained from user Y's calendar," but it is "schedule information obtained from user Y's email," and is an example of "user Y's schedule information."

[0166] A schedule "Lunch at Restaurant N from 12:00 to 14:30 on Sunday, March 14th" is a binding schedule that binds User Y between 12:00 and 14:30 on Sunday, March 14th. In contrast, the "Pickup period: one week from 3:00 PM on Sunday, March 14th" is a grace period in which you can perform a specific action at any time during this period. Whether a schedule is of the binding type or the deferred type may be set by the user, or alternatively, whether a schedule is of the binding type or the deferred type may be determined automatically based on the characteristics of the schedule. When making an automatic judgment, for example, plans that require reservations, such as meetings, meals, exercise, and classes, are judged to be binding plans. In addition, plans that have a set period, such as an application period or a pick-up period, are judged to be graceful plans.

[0167] [Leaving clothing store T] After leaving clothing store T, the user starts walking in the direction of restaurant N. The shortest time required to get from clothing store T to restaurant N is 15 minutes. The current time is 11:20, and there are 40 minutes remaining until the reservation time at Restaurant N, which is 12:00.

[0168] The prosperity levels of the areas around Clothing Store T, Shibuya Station, and Restaurant N are "Shibuya Station > Clothing Store T > Restaurant N." Restaurant N is located in a secluded area. The prosperity of a place can be calculated, for example, by the number of stores in the vicinity (for example, within a 50m radius); the more stores there are, the greater the prosperity. In this embodiment, the number of stores around restaurant N is less than a threshold value, and the prosperity level around restaurant N is less than a threshold value.

[0169] In this case, if user Y arrives at restaurant N too early, there will be few facilities nearby where he or she can kill time, and the user will end up with a lot of free time. For this reason, in this embodiment, more information about surrounding objects is presented to user Y when he is near clothing store T than when he is near restaurant N, and when he is near Shibuya Station than when he is near clothing store T.

[0170] In other words, the CPU 111 of the external server 100 determines the content of the surrounding object information to be presented to the user based on the prosperity level on the user's route, which is identified based on the user's schedule information. The CPU 111 of the external server 100 may determine the content of the surrounding object information to be presented to the user based on the degree of prosperity on the user's route identified based on the user's schedule information. The CPU 111 determines the content to be displayed on the display screen so that the more prosperous a location on the travel route is, the more information is presented to the user.

[0171] More specifically, in this embodiment, the more the distribution of objects that the user will pass on the way from the current location to the scheduled venue is biased toward the current location, the more surrounding objects are selected so that the user in the direction of the current location will be guided to. In addition, in this embodiment, the more the distribution of objects to be passed is biased toward the planned venue, the fewer surrounding objects are selected so that a user currently in the local area is guided.

[0172] The remaining time until the reservation time for restaurant N changes over time, and the required time, which is the time required to travel to restaurant N, also changes depending on user Y's behavior. For example, user Y may stay at a store on the route to restaurant N, or may deviate from the shortest route due to getting lost or making a detour, and the ratio of remaining time to required time may change accordingly.

[0173] If there is sufficient time remaining, more peripheral object information is presented, or the amount of information for each peripheral object information item is increased. On the other hand, when there is little time left, it is preferable to reduce the amount of peripheral object information to be presented or to stop presenting the peripheral object information. Furthermore, when there is little time left, it is preferable to provide route guidance so that user Y can arrive at the scheduled location by the scheduled start time.

[0174] According to the schedule information of user Y, user Y is scheduled to meet friend S at restaurant N. While user Y is on his way to restaurant N, he may be presented with nearby object information that matches his interests, but it is not limited to this, and user Y may also be presented with nearby object information that matches the interests of friend S, who is related to a future lunch plan.

[0175] An example of nearby object information that matches the interest of user Y is information such as coupons for stores that user Y frequently visits. Further, an example of information about nearby objects that match the interests of friend S is information about stores that sell brands that friend S is interested in.

[0176] For example, when user Y comes into view of a store of brand G, which friend S has registered as a favorite brand on a social networking site, information about the store of brand G may be presented to user Y. More preferably, information about the store of brand G and friend S may be presented in association with each other. For example, user Y may be presented with information such as, "Friend S, whom you are planning to meet soon, likes brand G." More preferably, user Y may be guided to the store of brand G by presenting a coupon for 10% off the gift set along with a suggestion to "use the coupon to buy a gift set of brand G for friend S."

[0177] The schedule information of user Y may include information about related people other than user Y, such as information about friend S. The CPU 111 presents to the user Y, for example, information about stores in which the related person is interested and stores that sell products in which the related person is interested, as nearby object information.

[0178] The interests and degree of interest of a particular person such as the above-mentioned friend S can be identified, for example, based on information registered as favorites on the friend S's SNS account. Additionally, the interests and the degree of interest of friend S may be identified based on friend S's activities on SNS, friend S's behavioral history, and the like. When identifying friend S's interests and level of interest based on friend S's activity on social media, for example, friend S's interests and level of interest can be identified based on text and photos posted by friend S, articles shared by friend S, and posts by people other than friend S that friend S has liked. In addition, for example, friend S's interests and level of interest may be identified based on friend S's location information, purchasing information, vital signs information acquired by the smart glasses worn by friend S, and information about friend S's field of vision acquired by the smart glasses.

[0179] User Y's next plan is to eat at restaurant N. This can be determined by analyzing the string "Lunch with Friend S" registered in User Y's calendar (schedule information), and also from the fact that the scheduled location is a restaurant called Restaurant N. In this case, even if information recommending Ramen Shop M is presented to User Y who is heading to Restaurant N, it is unlikely that User Y will stop by Ramen Shop M. It is preferable that the peripheral object information to be presented to user Y is determined based on the user's schedule information. More specifically, in this case, it is preferable that the peripheral object information to be presented to user Y is determined based on the user's schedule information and the content of the service provided by the peripheral object.

[0180] Although not explained above, User Y has just returned from a business trip the previous day and is tired. User Y's calendar for the last week (an example of past events) contains more than 10 long meetings. In other words, User Y's calendar for the last week contains multiple events.

[0181] In this embodiment, the user's fatigue level is estimated based on the user's schedule information (past schedule information), the user's vital signs, and the like. An example of the vital information of a user is information about sleep. Although not explained above, user Y fell asleep on a train traveling from Ikebukuro Station to Shibuya Station, and this sleep information was acquired as vital information.

[0182] In this embodiment, when the user falls asleep at a timing that is not originally planned, the user's fatigue level is determined to be high. The user's fatigue level may be determined using either the schedule information or the vital sign information, or a combination of these two pieces of information. Furthermore, information about the user's sleep may be obtained using, for example, a sleep analysis application installed on a smartphone or the like.

[0183] If it is estimated that the user Y has a high level of fatigue, the CPU 111 presents to the user Y, as nearby object information, information recommending shops that provide relaxation, such as massage shop A (see FIG. 20) and aroma shop K. In other words, the CPU 111 identifies that the user is in a predetermined state based on the schedule information, and then presents to the user, as surrounding object information, information about a specific surrounding object that corresponds to the predetermined state among multiple surrounding objects around the user's current location.

[0184] When schedule information and vital information obtained from the user's living body are used, CPU 111 determines the display content on the display screen based on this schedule information and vital information obtained from the user's living body. In other words, in this case, the CPU 111 identifies the user's fatigue level based on the schedule information and vital information obtained from the user's living body, and determines the display content on the display screen of the terminal device 200 based on this fatigue level. The display content may include, for example, information encouraging the user to use a service providing facility that provides a service to the user. Massage Parlor A is a store that provides massage services. Aroma Shop K is a store that sells products that provide relaxation, such as aroma candles.

[0185] The fatigue level may be determined based on both the schedule information and the vital information, or based on only one of the schedule information and the vital information. Here, as above, information about massage parlor A and aroma shop K is not included in the user's schedule information, and in this embodiment, based on the user's schedule information, information about facilities not included in this schedule information is presented to user Y.

[0186] Alternatively, the CPU 111 may identify the degree of the user's condition based on the schedule information, and may control the amount of surrounding object information to be presented to the user in accordance with the identified degree. Specifically, for example, the degree of fatigue of a user may be determined based on the number of meetings.

[0187] According to the map shown in Figure 20, massage parlor A and aroma shop K are located close to each other and can be within the user's field of vision at the same time. If there are multiple relaxation facilities in the user's vicinity, all of them or only some of them may be presented to the user. As one example, CPU 111 may determine whether to present information about massage parlor A and aroma shop K, or change the priority for presenting information about each facility, based on the congestion levels of each facility. As another example, CPU 111 may determine whether to present information about massage parlor A and aroma shop K, or change the priority for presenting information about each facility, based on the predicted length of stay at each facility. The congestion level of each facility may be determined based on the number of people actually present at the facility, the number of users on the same day of the week in the past, or the number of users during the same time period, or the reservation status of the facility.

[0188] Specifically, if CPU 111 determines that user Y, who is near massage parlor A, does not have enough time left to use massage parlor A, CPU 111 does not present information recommending massage parlor A or lowers the priority of presenting this information. As an example, if the average stay time at massage parlor A is calculated to be one hour from the history of past users and user Y only has 25 minutes left, CPU 111 determines that user Y does not have enough time to use massage parlor A. As another example, if the service menu provided by massage parlor A registered in the information processing system 1 of this embodiment is "30-minute massage for 1,500 yen" and user Y only has 25 minutes left, CPU 111 determines that user Y's remaining time (25 minutes) is not enough to use massage parlor A.

[0189] On the other hand, if the remaining time of users in the vicinity of aroma shop K is 25 minutes and the average stay time at aroma shop K is 20 minutes, user Y has enough time left to use aroma shop K. In this case, the CPU 111 of this embodiment determines to present information about the aroma shop K as the surrounding object information. As a result, in this example, information recommending aroma shop K is presented to user Y.

[0190] As described above, the CPU 111 acquires situation information, which is information about the situation of each of the surrounding objects (objects) present around the user. Specifically, as described above, the CPU 111 identifies, for example, the time required to use the facility as the status information of each of the surrounding facilities that provide massages and the like. The CPU 111 of this embodiment then takes this situation information into consideration when determining whether or not to include information about each facility in the surrounding object information. In other words, the CPU 111 of this embodiment determines whether or not to include information about each facility in the surrounding object information based on the schedule information and the situation information. As a result, as described above, information about aroma shop K will be displayed as nearby object information, while information about massage parlor A will not be displayed.

[0191] [Arrived at Park S, arrived at Restaurant N] As described above, the user Y was presented with nearby object information on the way from the clothing store T to the restaurant N. However, the user Y ended up not stopping at either store. After that, user Y took a nap on a bench in park S next to restaurant N until it was time to meet friend S. In this embodiment, information about this nap was registered in information processing system 1 by a sleep management application.

[0192] Even if the surrounding object information presentation function of the terminal device 200 of the user Y is enabled, there are cases in which the surrounding object information is not presented to the user Y. For example, there may be a case where there is no object around the user Y that can present information, or there is no object around the user Y that is determined to be the object that should present information to the user Y.

[0193] When the surrounding object information presentation function is enabled, the device continues to search for surrounding objects, which can lead to battery consumption and increased communication charges. Therefore, when certain conditions are met, the surrounding object information presentation function may be disabled. Specifically, for example, if a situation in which surrounding objects cannot be recognized continues for a certain period of time, or if the user continues to stay in the same place for more than a predetermined period of time, the surrounding object information presentation function may be disabled.

[0194] Later, User Y had lunch with his friend S at Restaurant N. User Y has installed a nutritional management application on his / her smartphone. While eating, User Y recorded the food using this nutrition management application. Specifically, User Y recorded the time when he ate and the food he ate. User Y also took a photo of the food that was served. User Y also posted a photo of the dessert on social media with the caption "Lunch with friend S." User Y's location information was attached to the photo and post.

[0195] The nutritional management application has a function of recording the calories and nutrients that user Y has taken in. The calories and nutrients ingested are identified, for example, by a nutrition management application analyzing a photo of food (an example of an image). Additionally, information about calorie and nutrient intake may be provided to the nutritional management application from a store or database.

[0196] [Departing from Restaurant N] When user Y leaves restaurant N, user Y is full. In this embodiment, the hunger level of user Y can be determined by using the following information (a) to (g) individually or in combination.

[0197] (a) User Y's hunger level can be determined based on information about the user's schedule, for example, a schedule "Lunch with friend S at restaurant N" registered in user Y's calendar. This is a method of determining hunger level based on schedule information. In other words, if user Y canceled this schedule but forgot to delete it from the calendar, determining user Y's hunger level based on user Y's schedule information may result in an incorrect determination. (b) Furthermore, based on the location information and the time spent at Restaurant N, which indicates that User Y stayed for 2 hours and 35 minutes, the level of hunger of User Y can be determined. This is a determination of hunger level based on events that actually occurred, not on schedule information. However, even if user Y is staying at restaurant N, the purpose of his / her stay may be something other than a meal, such as an interview, or restaurant N may be the user's workplace. When the determination is based on location information and stay time information, the hunger level of user Y is determined while ignoring the possibility that user Y may not have actually eaten a meal. (c) User Y's hunger level can be determined based on vital information related to hunger level, such as blood sugar level. This is a determination of hunger level based on events that actually occurred, rather than on scheduled information. However, when the hunger level is based on vital signs, the results of the hunger level assessment may be affected by medications or chronic illnesses. Also, when the hunger level is based on vital signs, changes in hunger level may not appear as changes in the vital signs until some time has passed since eating. (d) User Y's hunger level can be determined based on the food records in the nutrition management application of User Y. This is a determination of hunger level based on events that actually occurred, not on scheduled information. In addition, there is a risk that the user may forget to record meals in the nutrition management application or that the recording may be delayed from the time when the meal was actually eaten, and therefore the recorded information may not be immediately useful in determining the level of hunger of user Y. (e) User Y's hunger level can be determined based on posts made by the user on social media (for example, text such as "I ate this" and a photo of the food eaten). This is a determination of hunger level based on events that actually occurred, rather than on scheduled information. In this case, the user may not always post to the SNS, and may not post immediately during or after a meal, but may post at a later time. (f) User Y's hunger level can be determined based on the fact that points have been awarded to the membership card of Restaurant N for using the restaurant. This is a determination of hunger level based on actual events, not on schedule information. In this way, hunger level cannot be determined based on points in stores that do not use membership cards. Also, if a user forgets to present their membership card, points will not be awarded, so hunger level cannot be determined based on point award history. (g) User Y's hunger level can be determined based on the payment record to Restaurant N. More specifically, User Y's hunger level can be determined based on information registered in the restaurant's database, User Y's payment information, and information registered in a household accounting application. This is a determination of hunger level based on actual events, not on scheduled information. In this case, if a friend S other than user Y makes the payment or if the payment is made in cash, it will not be possible to determine user Y's hunger level.

[0198] The user's hunger level may be determined based on only one of (a) to (g), but each of the above determination factors (a) to (g) has drawbacks, and using only one of the determination factors (a) to (g) may result in a lower accuracy in determining the user's hunger level. For this reason, it is preferable to determine the hunger level based on multiple determination factors from among the determination factors (a) to (g). In particular, it is preferable to judge the hunger level using the above-mentioned (a) judgment element for making a judgment based on schedule information and one or more of the judgment elements (b) to (g). Furthermore, when using the determination factors (b) to (g), it is preferable to use a determination factor that determines the user's hunger level based on information actually acquired from the user's living body using sensor 213. Specifically, it is preferable to use the determination factor (c).

[0199] After that, User Y left Restaurant N and started walking. At this point, for example, if the surrounding object information presentation function and route guidance function are disabled, the surrounding object information presentation function and route guidance function will be enabled in response to detecting that user Y has started moving. User Y's next appointment is "Clothing store T, pick-up period: 3 / 14 (Sun) 15:00 - 1 week."

[0200] As described above, this appointment is a postponed appointment. Furthermore, when user Y leaves restaurant N, user Y is near the location of this appointment. Furthermore, in this embodiment, the CPU 111 searches for user Y's schedule up to 15:00 on Sunday, March 21st, and finds that the user has no plans to come to a location closer to clothing store T than their current location by March 21st, which is the pick-up period.

[0201] In this case, the surrounding object information presentation function of this embodiment performs processing to ask user Y, "The pick-up period at clothing store T starts today at 3:00 p.m. and runs for one week. If you miss today, it seems you won't be coming near clothing store T for a while. Would you like to pick it up later today?" If user Y answers "Yes," the surrounding object information presentation function asks user Y, "Is there anything else you would like to do around here?"

[0202] If user Y answers, for example, "No," the surrounding object information presentation function presents user Y with a message saying, "We will now show you information about deals at nearby facilities until you reach clothing store T," and the surrounding object information presentation function begins presenting surrounding object information. In this embodiment, the start time of the next appointment is 15:00, the departure time from restaurant N is 14:30, and the travel time from restaurant N to clothing store T is 15 minutes. Therefore, in this embodiment, the surrounding object information presentation function is enabled, and guidance using the surrounding object information presentation function is started.

[0203] At this time, the surrounding object information presentation function may be used to inquire of the user Y about the arrival time at the clothing store T. Specifically, the user Y may be inquired about the arrival time at the clothing store T by making an inquiry such as "When do you want to arrive at the clothing store T?". If user Y answers "about 10 minutes earlier than 3:00 PM" or "about 4:00 PM," the nearby object information presentation function presents nearby object information in accordance with this answer.

[0204] The reference time for the guidance of nearby objects may not be the "start time of the next appointment itself," but may be a time desired by the user, as described above. Some users may wish to arrive early because they have an important appointment, or may wish to arrive early to perform other tasks. In this case, by inquiring of the user about the time desired by the user, the reference time for the guidance of the surrounding objects can be made to correspond to the time desired by the user.

[0205] In addition, in the case of a postponed schedule, there is no need to arrive on time for the start time of the schedule. Therefore, in the case of a delayed schedule, the weight of the user's interest in the surrounding objects may be given more weight than the weight of the available time, and the surrounding object information may be presented. Furthermore, if the next schedule is a postponement-type schedule, processing may be performed so that more peripheral object information is presented than if the next schedule is a binding-type schedule.

[0206] [Stop by Bookstore E] The CPU 111 refrains from presenting information about restaurants to the user Y who is full. Furthermore, the area around Shibuya Station is not within the daily activity range of user Y. For this reason, user Y is given priority in being presented with information about surrounding objects that are not within the daily activity range, rather than information about surrounding objects that are within the daily activity range. Specifically, for example, the CPU 111 presents information about surrounding objects in which the user Y has shown interest while the user Y is on his way to the restaurant N.

[0207] However, without being limited to this, the CPU 111 may also present information about nearby objects that user Y showed interest in on the way to restaurant N, and that user Y also showed interest in on the way back. Additionally, for example, when user Y is on his way to restaurant N, he may be asked a question such as "Would you like to stop by on your way back?" in regards to a nearby object that user Y has shown interest in, to confirm whether user Y is truly interested. Then, information about surrounding objects that are of real interest to user Y may be presented to user Y on the way home.

[0208] Furthermore, the surrounding objects for which information is to be presented to the user Y may be determined based on information collected up to the present time, not limited to information obtained while the user Y is on his way to the destination. Although not explained above, User Y is a car enthusiast and always purchases Car Magazine A, which is released on the second Sunday of every month. Specifically, User Y purchases this magazine on the second Sunday or a few days later.

[0209] In this embodiment, the purchase history of this magazine is stored as "payment history@bookstore" in the payment application. The purchase history of this magazine is also stored as a purchase history in the household accounting application (for example, "X month Y day: Car magazine A (500 yen) @ Mizuno Bookstore"). In addition, the purchase history of this magazine is also accumulated as the bookstore's sales history (customer ID: yamada, sales date and time: February 14, 2020, 15:34:50, purchased item: car magazine A).

[0210] Today is the second Sunday, and the CPU 111 determines from the user Y's past behavioral patterns that there is a high possibility that the user Y will go to a bookstore, and presents the user Y with information about bookstores. In other words, the CPU 111 determines that the user Y is likely to visit a bookstore based on the purchase history and sales history, and presents the user Y with information about bookstores. Specifically, the CPU 111 searches for bookstores that have the item that the user Y is interested in purchasing (car magazine A in this example) in stock, and presents the bookstores that have the item in stock to the user Y.

[0211] In this processing example, the book is in stock at bookstore E, and information about bookstore E is presented to user Y. Specifically, when bookstore E comes within the user's field of vision, information such as "Car magazine A on sale" is presented to user Y. In this processing example, user Y then reacts to this information by going into bookstore E and purchasing a magazine called car magazine A. User Y also looks through other magazines that were located near car magazine A.

[0212] [Leaving Bookstore E] After that, User Y left Bookstore E. In this case, the CPU 111 lowers the priority of the bookstore. Specifically, in this case, it is considered that the purpose of visiting the bookstore has been achieved since the user Y has already purchased the magazine, so the CPU 111 lowers the priority of the bookstore. As a result, when user Y passes by a bookstore other than bookstore E in the future, that bookstore will not be selected as a nearby object for which nearby object information will be presented.

[0213] If it is confirmed that user Y has visited bookstore E and then entered another bookstore, CPU 111 may again increase the priority of the bookstore. In this case, it is possible that user Y is, for example, visiting bookstores, so by increasing the priority of bookstores, information that is useful to user Y is presented to user Y. In this way, if a user has entered a particular type of store (e.g., bookstore, shoe store, etc.) multiple times, the presentation of this particular type of store may be given a higher priority.

[0214] [Arriving at clothing store T. Picking up the suit after hemming is complete.] After that, User Y arrives at Clothing Store T and picks up the suit with the hemming completed. After that, User Y receives an email saying "Thank you for picking up the suit." With this, although the scheduled end time of "Pickup period: 1 week from 15:00 on Sunday, March 14th" has not yet arrived, this schedule is considered to have already been completed.

[0215] [Leaving clothing store T] After that, User Y walked towards Shibuya Station. No events for User Y were retrieved from the calendar, email, or reservation system for the rest of the day. In other words, even if information was extracted and analyzed from calendars, emails, reservation systems, etc., no schedule information for User Y was extracted. In this case, it is not a situation where User Y is "killing time until the next appointment."

[0216] It is thought that there are users who "want to get home early" and users who "do not want to get home early." Places where users who "want to get home early" are likely to visit are supermarkets and stores that sell daily necessities, etc. Users who "want to get home early" are unlikely to visit cafes, restaurants, brand stores, etc. Whether user Y likes to go home early or not can be determined by analyzing past behavioral information. Specifically, whether user Y likes to go home early or not can be determined based on, for example, whether user Y tends to make a detour after the last event of the day.

[0217] The CPU 111 considers whether the user is on the way home. Specifically, since the user Y is moving along the route to his / her home, the CPU 111 determines that the user is on his / her way home. In this case, the CPU 111 makes a decision about presenting information to the user, taking into consideration that the user is on the way home.

[0218] On the way home, the user boards a train at Shibuya Station and travels towards Ikebukuro Station. The CPU 111 takes into consideration the user's wishes, such as not wanting to get on the train with luggage, or wanting to buy food and daily necessities after arriving at the nearest station to their home. Furthermore, if the user gets home late, there is a possibility that the supermarket near the user's nearest station will close while the user is on the move. In this case, it is preferable to compare the closing time of a place the user frequently visits (for example, a supermarket near the nearest station) with the time the user plans to pass by, and if the supermarket near the nearest station will be closed by the time the user arrives, to present the user with information about supermarkets near Shibuya Station as nearby object information when the user passes by a supermarket near Shibuya Station.

[0219] For example, if a user is walking around Shibuya Station and passes a supermarket, the system will present the user with information such as, "Do you need to buy some groceries? There is a supermarket on your way home, but it will probably be closed by the time you get there. We recommend that you buy your groceries here."

[0220] [User Y arrives at Shibuya Station and takes the train from Shibuya Station to Ikebukuro Station] In this case, for example, the surrounding object information presentation function is disabled, as in the case of the outbound journey. Furthermore, when the train arrives at Ikebukuro Station and it is detected that the user has gotten off the train, the surrounding object information presentation function is enabled. More specifically, when it is detected that the user has boarded a train, the surrounding object information presentation function is disabled, and when it is detected that the user has disembarked from the train, the surrounding object information presentation function is enabled.

[0221] [Arriving at Ikebukuro Station and starting to walk] After that, User Y arrived at Ikebukuro Station (see FIG. 21) and started walking. In this embodiment, as described above, the surrounding object information presentation function is enabled in response to detection that the user has gotten off the train. Here, since the user frequently travels between the nearest station and his / her home, there is usually no need to provide route guidance information between the nearest station and his / her home. Therefore, in this case, the CPU 111 does not provide route guidance information.

[0222] However, in this embodiment, if the user frequently purchases items on the way home or frequently stops at a store, the CPU 111 may present information when the user passes by a store that sells these frequently purchased items or this frequently visited store. Specifically, the CPU 111 may present to the user, for example, advantageous information about the store as nearby object information. The advantageous information may include, for example, sale information about the store.

[0223] Furthermore, when a specific condition is met, the CPU 111 guides the user Y along a route that the user Y does not usually take, thereby expanding the user Y's range of movement. For example, when the user has completed the last appointment within a predetermined period (for example, when the user has completed the last appointment of the day), the CPU 111 suggests that the user take a route that the user does not usually take.

[0224] More specifically, in this case, CPU 111 determines whether the current time is before the last scheduled event of the user within a predetermined time limit. Then, the CPU 111 guides the user along different routes depending on the result of this determination. If the determination result is, for example, that the current time is later than the last scheduled time within the user's predetermined period, the CPU 111 suggests to the user to take a route that is not usually taken. On the other hand, if the result of the determination is that the current time is before the last scheduled time within the user's predetermined time limit, for example, the CPU 111 does not suggest taking an unusual route.

[0225] User Y takes the shortest route from the nearest station (Ikebukuro Station) to his home almost every day. By the way, a new bakery called Bakery K recently opened on a road just off the road that User Y usually takes. Additionally, based on User Y's purchasing history and social media posts, it was discovered that User Y likes bread.

[0226] In this case, the CPU 111 suggests to the user Y that he or she take a route that he or she normally does not take by presenting the user with a message such as, "A bakery called Bakery K opened yesterday about a two-minute walk from here. You can stop by Bakery K in about five minutes longer than the shortest travel time to your home. Would you like to go there? If you want to check the location on a map, please look at your smartphone."

[0227] Since the route to Bakery K is a route that User Y does not usually take, if the user proceeds along the suggested route (for example, if User Y answers, "I want to go to Bakery K"), CPU 111 may enable the route guidance function and provide guidance on the route to Bakery K. At this time, while providing guidance on the way to the bakery K, the CPU 111 may present to the user, as nearby object information, information about stores along the route that the user Y will take to get to the bakery K.

[0228] Additionally, the CPU 111 may provide guidance to the user Y on the route that the user Y usually takes from various viewpoints. Here, user Y is 30 years old, male, married, has a child (for example, a 3-year-old son), and is a manager at company X. When user Y wants to know what other "men in their 30s" are looking at as they walk around, CPU 111 controls the display on the display screen in, for example, "men in their 30s mode."

[0229] Additionally, for example, guidance to user Y can be given in "user Y's wife mode." In this case, for example, the user is informed of things that the wife of user Y usually notices. Information about the things that usually notices can be acquired by analyzing images captured by the camera 214 of the terminal device 200 that the wife usually wears. The user Y is guided to the things that the wife of the user Y usually notices, for example, in response to an instruction from the user Y.

[0230] Furthermore, for example, the birthday of user Y's wife may be identified based on information registered in user Y's calendar (an example of schedule information), and user Y may be guided in "user Y's wife mode" in the month before the wife's birthday, for example. This allows user Y to obtain information about a store that his wife is interested in, and he can, for example, purchase a present for his wife at this store.

[0231] Also, for example, a mode called "President Mode" may be prepared, and when this President Mode is selected, information about surrounding areas that are of interest to people with the title of President may be presented to User Y. If user Y has ambition, presenting such information to the user will make the information useful to user Y.

[0232] [Arrived at Bakery K and bought some bread.] After that, User Y arrives at Bakery K and purchases bread there. In this case, in this embodiment, the CPU 111 asks the user Y, "Would you like to introduce Bakery K to me?" and prompts the user Y to post on the SNS. Specifically, for example, when user Y leaves the store, the user is prompted to post on SNS.

[0233] Alternatively, the user Y may be asked to speak while he or she is on the move after leaving the bakery K, and the text to be posted on the SNS may be created based on the user's voice. In this case, the user Y can save time. Additionally, the user may be prompted to process still images or videos captured by the camera 214 of the terminal device 200 (for example, by cropping or mosaic processing) and post the processed still images or videos.

[0234] Alternatively, a product that appears in the video captured by the camera 214 of the terminal device 200 for a long period of time may be identified, and the portion of the video in which the product appears may be cut out as a still image or video. This product is considered to be of great interest to user Y. Furthermore, to protect personal information, still images and videos may be processed so that people and their voices are not included. Furthermore, the CPU 111 may ask the user Y, "Would you like to rate Bakery K?", and request the user Y to rate the bakery.

[0235] Furthermore, after the user Y leaves the bakery K, the CPU 111 may ask the user Y, "Are you going home?". If the user Y answers "yes," the navigation system provides route guidance to the user Y so that the user Y can reach the route that the user Y normally takes to return home from Ikebukuro Station. In this case, the navigation system may end when the user Y reaches the route that the user Y normally takes.

[0236] Depending on the type of store and the service received by user Y, it may be difficult for user Y to rate the store immediately after leaving it. For example, if you take out bread from Bakery K, it is possible to evaluate the service and the atmosphere of the store immediately after leaving, but it is difficult to evaluate the taste of the bread immediately after leaving the store because you have not yet eaten it.

[0237] Therefore, the user will rate the bread after some time has passed since the user left the store, i.e., three days or one week later, when User Y has definitely eaten the bread. Furthermore, if the user is asked to rate the service etc. after they have definitely eaten the bread (such as three days or a week later), there is a risk that the user's memory of the service or the atmosphere of the store may have already faded. For this reason, it is preferable to request the user to evaluate the customer service and the atmosphere of the store immediately after the user visits the store.

[0238] Here, the evaluations may include inauthentic evaluations. Inauthentic reviews include, for example, reviews of places that have not actually been visited, reviews of products by users who have never used or eaten the product, and reviews of customer service by users who have not received the service.

[0239] The determination of whether a user's evaluation is genuine may be made based on video and audio information acquired by the user's terminal device 200, by determining whether the user actually looked at the product or whether the user actually communicated with the store clerk.

[0240] The image captured by the user's terminal device 200 is, for example, an image captured by the camera 214 of the terminal device 200. Based on this image, what was in the user's field of view is confirmed, and a determination is made as to whether the user's evaluation is genuine or not. Furthermore, the voice acquired by the user's terminal device 200 is, for example, voice acquired by the microphone 215 of the terminal device 200. By analyzing this voice, for example, the content of the conversation between the user and the store clerk can be identified, and based on the identification result, it is determined whether the user's evaluation is genuine or not.

[0241] In addition, based on the video and audio information acquired by the user's terminal device 200, users who were clearly viewing the product or communicating with the store clerk can be identified, and fraudulent evaluations can be avoided by asking the identified users to evaluate the content that they are able to evaluate. Specifically, for example, a user who was communicating with a store clerk may be asked, "How was the customer service?". Also, for example, a user who was viewing products may be asked, "What was the product lineup like?", "Did you find any products you wanted?", etc.

[0242] Here, the determination of whether or not the user actually visited the store can be made based on, for example, the length of time the user stayed in the store. In this case, however, even if the user is sitting in a chair inside the store or simply passing through the store, there is a risk that the user will be determined to have actually visited the store. In contrast, when making a judgment based on video and audio information, as described above, evaluations can be requested from users who have definitely seen the product or who have communicated with the store clerk.

[0243] Additionally, the content of the evaluation requested from the user may be changed based on the user's behavior in the store. Specifically, for example, a user who uses a self-checkout and does not receive any customer service may be asked to rate elements other than the customer service. In this case, it is less likely that a problem will occur in which a customer answers whether the customer service was good or bad even though the customer has not actually received the service, resulting in a decrease in the accuracy of the evaluation.

[0244] Alternatively, whether the content of the user's evaluation is authentic may be determined based on data acquired by the terminal device 200 of the user. For example, if a user gives an evaluation such as "I was very annoyed when the store clerk called me 'XXX'," it may be possible to determine whether or not this evaluation is genuine based on the voice information acquired by the user's terminal device 200.

[0245] Also, for example, if an evaluation such as "I noticed a scratch on the product when I got home" is made, it may be determined whether or not this evaluation is genuine based on the video information and acceleration information acquired by the user's terminal device 200. Specifically, in this case, the behavior of the user after the user leaves the store is analyzed based on the video information and acceleration information acquired by the user's terminal device 200.

[0246] For example, if the user's behavior after leaving the store is such that the user makes violent movements that damage the merchandise, it is determined that the user's evaluation may not be genuine. When such processing is performed, it is possible to prevent evaluations from being made based on events that have not actually occurred.

[0247] [I enter Supermarket F to buy ingredients for dinner.] Afterwards, User Y went into Supermarket F and bought ingredients for dinner. As mentioned above, information about fresh ingredients for dinner is preferably presented to the user on their way home. For this reason, in this embodiment, information about fresh ingredients to be used for dinner is presented to user Y when user Y stops by supermarket F on his / her way home and supermarket F comes into user Y's field of vision.

[0248] [Returning home] After that, the user Y arrives at his / her home. In this case, the CPU 111 displays the information "Thank you for your hard work!" on the terminal device 200 in response to the user Y's arrival at his / her home, and then ends the navigation. In other words, the CPU 111 disables the route guidance function and the surrounding object information presentation function.

[0249] After that, the CPU 111 may ask the user Y how satisfied he or she is with the navigation. Specifically, the CPU 111 makes inquiries such as "Are you satisfied with the navigation?" and "Was the amount of information appropriate?". If the user replies that he / she is not satisfied or that the amount of information is not appropriate, the CPU 111 modifies the content of the processing to be performed thereafter in accordance with the content of this reply.

[0250] Furthermore, the CPU 111 may make an inquiry such as, for example, "Was the information accurate?" If the user replies that the information was not accurate, the CPU 111 performs processing to correct the information. Furthermore, the CPU 111 makes an inquiry such as, "Was the necessary information provided?" If the user replies that the necessary information was not provided, the CPU 111 corrects the process related to the presentation of information in accordance with the user's reply.

[0251] [Modification] In the above, an example has been described in which peripheral object information, which is information about peripheral objects around the current location of user Y, is presented to user Y at this current location. Here, a process is described in which, instead of surrounding object information, related object information about related objects that have a predetermined relationship with surrounding objects in the vicinity of the current location of user Y is presented to user Y at the current location. As an example, similar objects that have something in common with the surrounding objects are identified, and information about the similar objects is presented as related object information.

[0252] In the above, an example of a process has been described in which, when a store called Aroma Shop K (see FIG. 20) comes into view of user Y who is moving around, information recommending Aroma Shop K is presented to user Y. In this embodiment, as shown in FIG. 20, 30 meters away from aroma shop K, there is another aroma goods store called aroma shop U, which provides almost the same services as aroma shop K. When user Y is in front of aroma shop K, aroma shop U is not in the field of view of user Y. Aroma shop U is on the route that user Y takes when heading to restaurant N (an example of the location of the next appointment). Aroma Shop U and Aroma Shop K are service facilities that share a commonality in that they both provide aroma-related services.

[0253] Furthermore, in this embodiment, aroma shop K does not pay a promotion fee to the operator of the information processing system 1 of this embodiment, whereas aroma shop U does pay a promotion fee. In other words, in this embodiment, the operator of aroma shop K does not pay a fee to have information about aroma shop K presented. In response, the operators of Aroma Shop U pay a fee to have information about Aroma Shop U displayed. It should be noted that payment of the fee is not mandatory, and information about Aroma Shop U may be presented even if the fee is not paid.

[0254] In the processing example of this embodiment, the user Y passes through the aroma shop K before passing through the aroma shop U. In this case, if user Y buys what he needs at aroma shop K, he is likely not to stop by aroma shop U. In contrast, when information about aroma shop U is presented to user Y who is in the vicinity of aroma shop K, as in this embodiment, the possibility of user Y stopping by aroma shop U increases.

[0255] Examples of information about aroma shop U presented to user Y include information about the location of aroma shop U (for example, information such as "30 meters away") and information encouraging use of aroma shop U, such as discount coupons. By presenting such information, the possibility that user Y will shop at aroma shop U instead of at aroma shop K increases. Alternatively, if such information is presented, the user may check the prices and product lineup at Aroma Shop K, but may also want to check out Aroma Shop U before purchasing the product.

[0256] Even when presenting information about aroma shop U, if aroma shop U is not on user Y's route, the effect of presenting information about aroma shop U to user Y is diminished. For this reason, it is preferable to determine whether or not information about the aroma shop U should be presented to the user Y, taking into consideration the schedule information of the user Y. For example, it is preferable to determine whether or not information about the aroma shop U should be presented to the user Y, based on information about the location to which the user Y is planning to go, which is included in the schedule information of the user Y.

[0257] In this embodiment, based on the schedule information of the user Y, the route of the user Y when the user Y heads to the scheduled location is predicted. In this embodiment, an object similar to the surrounding object and located on the user's route is selected. In other words, an object related to the surrounding object that has a predetermined relationship with the surrounding object and is located on the route is selected. In this embodiment, it is determined that the business type of aroma shop K and the business type of aroma shop U are the same based on information registered in the database. In this embodiment, when the user visually recognizes aroma shop K, aroma shop U, which is in the same business as aroma shop K, is selected.

[0258] When presenting similar objects, it is preferable to present similar objects that are within a predetermined distance from the surrounding objects. Furthermore, a user Y viewing aroma shop K may be presented with both information about aroma shop K and information about aroma shop U. Also, for example, the price of a product that user Y wants at aroma shop U may be compared with the price at aroma shop K, and information about the store with the lower price may be presented to user Y.

[0259] Specifically, for example, if the price at Aroma Shop U (which may be, for example, a guideline price for using Aroma Shop U (for example, an average price) or the price of a specific product) is lower than the price at Aroma Shop K, information about Aroma Shop U may be presented to User Y. Alternatively, the evaluation of aroma shop U may be compared with the evaluation of aroma shop K, and if the evaluation of aroma shop U is better, information about aroma shop U may be presented to user Y.

[0260] In addition, the function of presenting information about similar products may be able to be switched on and off, and when this function is enabled, information about similar products, such as information about Aroma Shop U mentioned above, may be presented. Additionally, the user may be allowed to set conditions under which information about similar objects is presented. Specifically, for example, the user may be allowed to set conditions such as "display information about similar objects when there are similar objects with higher ratings."

[0261] [Device interaction] Additionally, information such as coupons that can be used in nearby locations, which is displayed on the glasses-type terminal device 200, may be sent by email or the like to another terminal device 200 (for example, a smartphone-type terminal device 200 or a wrist-worn terminal device 200). In this case, coupons and the like displayed on the eyeglass-type terminal device 200 can be displayed on other terminal devices 200 and saved.

[0262] The transmission of information about surrounding objects to other terminal devices 200, the display of information about surrounding objects on other terminal devices 200, and the storage of information about surrounding objects on other terminal devices 200 may be performed when the user gives a voice instruction or a gesture instruction. Alternatively, this may be done when a hardware button or software button is selected by the user.

[0263] For example, when the user reacts to the presented surrounding object information, information such as a coupon corresponding to the reacted surrounding object information is transmitted to the smartphone-type terminal device 200 or the wrist-worn terminal device 200. More specifically, when a user looks at the presented surrounding object information or the surrounding object itself corresponding to the presented surrounding object information for a certain period of time or more, or when a visit of the user to this surrounding object is detected, a coupon that can be used at this surrounding object is sent to a smartphone-type terminal device 200 or a wrist-worn terminal device 200. In this case, the coupon or information indicating that a coupon is available is displayed on the display screen of the smartphone-type terminal device 200 or the wrist-worn terminal device 200.

[0264] For example, in the case of a coupon that scans a barcode or a QR code (registered trademark), it is difficult to present the coupon displayed on the eyeglass-type terminal device 200 to a barcode reader or a store clerk. In contrast, when a coupon is displayed on a smartphone-type terminal device 200 or a wrist-worn terminal device 200 as described above, the coupon is more likely to be presented and scanned. In addition, to ensure that the user does not forget to present the coupon when paying, when the user attempts to make an electronic payment, a message such as "Coupon available" may be displayed on the terminal device 200 that the user is attempting to use for payment or on other terminal devices 200.

[0265] [Other specific examples] Next, an example of the process will be described using a private booth (hereinafter simply referred to as a "booth"), which is an example of a rental space, as an example of a facility. The booth in this embodiment is installed in the city and has an entrance and an enclosure that users can enter through the entrance. By using the booth, users can work in their spare time. Users can reserve this space in advance, and the usage fee is set, for example, in specific time units (for example, 300 yen for 15 minutes).

[0266] In this processing example, a decision is made as to whether to invite a user to a booth based on the user's schedule information. In other words, a decision is made as to the presentation of booth information based on the user's schedule information. More specifically, a decision is made as to whether to invite a user based on the user's next schedule. As an example, the start time of the user's next appointment is used to determine whether to invite the user, and more preferably, the current location, the current time, and the start time and venue of the user's next appointment are used to determine the user's level of urgency and to determine whether to invite the user. In this processing example, for example, people who have enough time to use the service can be invited, and people who do not have enough time to use the service can be not invited. Examples of people who have enough time to use the service include people whose next event is scheduled within a predetermined period (for example, from the current time to the end of the day) and people who have more than a predetermined amount of time for the location and start time of their next event. Examples of people who do not have enough time to use the service include people whose next event is not scheduled within a predetermined period and people who do not have more than a predetermined amount of time for the location and start time of their next event.

[0267] As another example, whether or not to invite a user to a booth may be determined depending on the type of the user's next plan. For example, if the user's next appointment is a private appointment (eg, lunch with a friend), the user is not invited, but if the user's next appointment is a business appointment (eg, a meeting), the user is invited.

[0268] In addition, for example, based on the user's schedule information, the benefits or disadvantages to the surrounding object if the user uses the surrounding object may be determined, and a decision may be made regarding the presentation of information about the surrounding object to the user based on this benefit or disadvantage. Specifically, for example, based on the start time of the booth's next reservation and the location and start time of the next appointment of a user who passes by the booth, the benefit or disadvantage to the booth if the user uses the booth may be determined, and a decision may be made regarding the presentation of information about the booth to the user based on this benefit or disadvantage. More specifically, in this case, for example, the system first predicts the usage fee (an example of the benefits or disadvantages of the booth) for a user who uses the booth based on the start time of the next reservation for the booth and the location and start time of the next appointment of a user who passes by the booth.Then, based on this usage fee, it decides whether or not to invite the user.If it is decided to invite the user, it presents a proposal to invite the user.

[0269] Specifically, in this case, if the usage fee exceeds a predetermined threshold, the user is invited, and if the usage fee does not exceed the predetermined threshold, the user is not invited. If the user does not accept the invitation, a new user is searched for. In other words, if the booth is out of the field of view of the user who made the invitation, a new user is searched for.

[0270] In order to increase income from booth rental, it is preferable to give priority to soliciting people who are likely to use the booth for a long period of time. Considering the start time of a user's next appointment and the travel time to the next appointment location, it is preferable from the perspective of booth revenue to invite users who have the potential to use the booth for 30 minutes rather than inviting users who can only use the booth for a maximum of 15 minutes.

[0271] Alternatively, profits may be predicted by predicting, for each user, which services provided by a facility are likely to be used by the user.Furthermore, the amount to be paid by the user to the service providing facility (an example of a profit or loss for the service providing facility) may be predicted by predicting the cost of providing the service.

[0272] To explain this with a specific example, for example, it can be predicted that a user who had lunch plans until just now will only buy a drink. For example, if a user mentions on social media that they want to get a perm, it can be predicted that they will get a perm, and if a user mentions on social media that they want to get a haircut, it can be predicted that they will get a haircut. In addition, when determining the profit or loss of a service providing facility, the costs involved in providing the service may be taken into consideration. The costs involved in providing the service include the cost of purchasing products and materials, labor costs at the service providing facility, and rent for the space used to provide the service. For example, it is generally said that haircuts have a higher profit margin because they can be performed in less time than perms, resulting in lower labor costs and no costs for chemicals, etc. Therefore, if the surrounding facility is a hair salon, when soliciting users, it is preferable to solicit users who want to have their hair cut.

[0273] Also, if the surrounding property is a service facility such as a restaurant, by prioritizing attracting users who finish their meal and drink in a short time and leave, or users who take out food and drink, the restaurant's turnover rate will increase (an example of a benefit or disadvantage), and by providing food and drink to more users, the restaurant will be able to increase its profits. For example, if, based on the user's next schedule, there is a user who is likely to stay for up to two hours and another user who is likely to stay for up to one hour, if the former user uses the restaurant, there is a risk that that person will occupy a seat for two hours, which will reduce the restaurant's turnover rate.However, this risk can be avoided by inviting the latter user.

[0274] Furthermore, if users who are around a certain object and are with other people are invited to use the service provided by the object, there is a high possibility that multiple people will use the service, which is often beneficial for the service provider (an example of a benefit or disadvantage). In other words, it is preferable to present information about the object to users who are around the object and are with other people. Specifically, the system searches for users who are traveling with others, such as users on a date, and invites these users preferentially over users who cannot be assumed to be traveling with others. Whether a user is traveling with others can be determined, for example, from the schedule registered in the user's calendar.

[0275] Furthermore, inviting users who are with other people increases the utilization rate of two-seater and four-seater table seats, which is beneficial for the service provider (an example of a benefit or disadvantage). If you invite a single user who is not with other people, they will end up using a two- or four-seater table by themselves, which will result in empty seats and a lower utilization rate. In contrast, seat utilization rate can be increased by inviting users who are presumed to be with other people, especially users who are part of a group of an even number of people (for example, a group of two or four people). Note that this embodiment does not prevent inviting groups of an odd number of people, and if there is a three-seater seat, seat utilization rate can be increased by inviting a group of three people.

[0276] Therefore, in this embodiment, the CPU 111 determines, based on the user's schedule information, the benefits or disadvantages to a facility when the user uses the facility. As an example, the CPU 111 calculates the predicted usage time of the facility for this user. Then, based on this predicted usage time, the CPU 111 makes a decision regarding the presentation of information about the facility to the user.

[0277] As another example, decisions regarding the presentation of facility information to a user may be made based on the people associated with the user's schedule. Specifically, for example, if the number of people related to the user's schedule is an even number, CPU 111 presents information about a certain facility to the user, and if the number of people related to the user's schedule is one, CPU 111 does not present information about that facility to the user.

[0278] Additionally, when a user to be solicited passes through an affiliated store that is located before the booth and has an affiliated relationship with the booth, the user may be presented with a coupon that can be used at the affiliated store. More specifically, in this case, a coupon that is valid on the condition that the user uses the booth may be presented to the user who passes by the affiliated store. More specifically, a coupon that is valid on the condition that the booth is used may be displayed on the terminal device 200 of a user who passes in front of an affiliated store.

[0279] Additionally, when a user to be solicited passes by a booth that is being used by another user, the user may be presented with information about ways to kill time until a booth becomes available and information encouraging the user to reserve the booth. Alternatively, whether or not to invite a user may be determined depending on the user's level of interest in the booth. The content of the information to be presented to the user may also be determined based on the user's level of interest in the booth. Specifically, for example, if the user has a high level of interest in a booth, the user may be presented with the booth's reservation status, information encouraging reservation, and an explanation of the booth.

[0280] [Booth Case 1] The processing for the booth will be explained using a specific example. User B, who has a meeting scheduled at a client's office starting at 1:00 PM, passes by Booth X. The current time is 12:35. It will take just under five minutes to walk from User B's current location (in front of Booth X) to the location of the next scheduled meeting. In this case, the longest time that user B can use booth X is 20 minutes from 12:35 to 12:55. Booth X is currently vacant. Also, there are no booth reservations between now and 3:00 PM.

[0281] User B's next appointment is a "meeting," which is a work appointment. Booth X's service is to provide a space for work. Booth X can be used for preparing for or studying before a meeting, and the content of the service provided by Booth X is a good match with the content of User B's next schedule. In other words, there is a predetermined relationship between the content of the service provided by Booth X and the content of User B's next schedule. In this case, the CPU 111 invites the user B.

[0282] In other words, in this processing example, CPU 111 identifies that booth X exists in the vicinity of the user, and based on schedule information, which is information about the user's schedule, CPU 111 makes a decision regarding presenting information about booth X to the user. In this example, the CPU 111 determines to present information about booth X to the user. Here, as above, booth X is an entity that is not explicitly stated in the user's schedule information. In other words, booth X is an entity that is not included in the user's schedule information.

[0283] When CPU 111 determines to present information about booth X to the user, it controls so that information about booth X is presented to the user. Furthermore, when the CPU 111 determines not to present information about booth X to the user, it controls so that information about booth X is not presented to the user. If the content of the next plan for user B is a "date," CPU 111 will not invite user B.

[0284] In this embodiment, the CPU 111 makes a decision regarding the presentation of information about Booth X to the user based on the type of the user's schedule. In this embodiment, the schedule information includes information about the name, content, and type of the schedule. Specifically, in this embodiment, the schedule information includes information about the type of the schedule, such as "meeting." Based on this schedule type, CPU 111 makes a decision regarding the presentation of information about booth X to the user.

[0285] In this embodiment, as will be described later, information about related objects that are related to the surrounding objects that exist around the user may also be presented. When presenting information about related objects that are related to the surrounding objects that exist around the user, the presentation of this information may be determined based on the type of schedule of the user.

[0286] As another example, if there is another user, user E, who is nearby booth X and who may be willing to use the booth for 40 minutes, user E may be invited to use the booth rather than user B. Furthermore, when inviting user B, it may be determined whether or not to invite user B based on user B's past use trends of the booth.

[0287] Here, for example, it is assumed that when referring to the past booth reservation trends of user B, it is found that user B has almost never reserved booth X for less than an hour. In the example of this embodiment, as described above, the time that user B can use booth X is 20 minutes, which differs from user B's past booth usage trends. In this case, the CPU 111 may not invite user B.

[0288] (Other processing examples) Let us consider a case where a hair salon owner is soliciting users who are in the vicinity of the hair salon he or she manages. In this case, even if a user whose next appointment is to go to a "hair salon" is solicited, it is expected that the user will not respond well. In other words, in this case, the solicitation will be like soliciting a user who is about to receive a service from a competitor, and it is expected that the user will not respond well. Users who have a "hair salon" scheduled for their next appointment often already have a reservation at their regular hair salon. Also, it is unlikely that users who have a "hair salon" scheduled for their next appointment will go to the trouble of canceling and paying the cancellation fee to go to a new hair salon.

[0289] For this reason, in this case, it is preferable to solicit users who have enough time to receive a treatment at a hair salon and who have plans that day that will motivate them to dress up, such as a "date" or "lunch with a friend," because such users have a strong motivation to go to a hair salon. Also, users who have not had a scheduled visit to a hair salon for a long time will have an incentive to go to a hair salon, and it is preferable to solicit such users.

[0290] [Booth Case 2] User N, who has a meeting scheduled at a client's office starting at 5 p.m., passes by Booth X. The current time is 15:50. It will take 10 minutes for User N to travel from his current location to the location of his next scheduled meeting. In this case, the longest time User N can use Booth X is 60 minutes, from 15:50 to 16:50. Booth X is reserved by someone else until 4:00 PM, so User N cannot immediately begin using Booth X.

[0291] In this case, the CPU 111 of this embodiment presents to a user N who is passing by booth X, for example, information recommending a reservation at booth X from 16:00 as information about booth X, which is a nearby object, and presents a coupon for a service providing facility near booth X as information about related objects related to booth X. More specifically, in this example, the CPU 111 makes a decision regarding the presentation of information about booth X to the user based on the schedule information acquired about the user (e.g., the location, start time, and type of the user's next schedule) and booth status information that indicates the status of the booth (such as the booth's reservation status and usage status). Furthermore, based on the schedule information acquired about the user and the booth status information that indicates the status of the booth, the CPU 111 makes a decision regarding the presentation of information about service affiliated facilities near booth X to the user. In this example, the CPU 111 determines to present to the user N information recommending a reservation for booth X from 16:00 and a coupon for a service providing facility near booth X.

[0292] In this case, when selecting a nearby service providing facility, it is preferable to take into consideration the length of time from the current time until 16:00 when User N will be able to use Booth X, and the time it takes to travel from User N's ​​location (i.e., the location of Booth X) to and from each service providing facility. In this case, a nearby service providing facility may be, for example, a convenience store, and a coupon for a nearby service providing facility may be, for example, a coupon for a free coffee.

[0293] [Booth Case 3] User H, who has a client meeting starting at 5 p.m., passes by a service facility affiliated with Booth X. This service facility is located a one-minute walk from Booth X. The current time is 15:47. It takes 10 minutes to travel from Booth X to the location of the conference, which is the location of User H's next appointment. In this case, the longest time User H can use Booth X is 62 minutes, from 15:48 to 16:50. Booth X is reserved by someone else until 16:00, and if user H walks around without stopping anywhere, user H will pass by booth X at 15:48. In this case, it is predicted that booth X will still be in use by someone else.

[0294] In this case, the CPU 111 presents the user with, for example, information recommending the use of a service facility affiliated with booth X and information recommending a reservation for booth X from 16:00. If there are multiple affiliated facilities affiliated with Booth X, the affiliated facility to which the solicitation will be made is selected from among these multiple affiliated facilities.

[0295] Specifically, when selecting an affiliated facility from among multiple affiliated facilities, the system selects an affiliated facility where the user can kill time based on the estimated time required to receive services at each affiliated facility (calculated based on the stay time, waiting time, treatment time, etc.) and the time from the current time until the booth reservation start time. Additionally, the affiliated facility to be combined with the reservation of booth X may be selected taking into consideration the user's level of interest in the affiliated facility.

[0296] FIG. 22 is a diagram showing another example of display on the terminal device 200. In FIG. 22 shows an example of the terminal device 200 in which information about objects around the user is displayed on a smartphone-type terminal device 200. This display example shows a display example on the terminal device 200 when coffee shop S is within the user's field of view. In this embodiment, in this case, the CPU 111 identifies that a coffee shop S exists in the vicinity of the user, as in the above case.

[0297] In this case, the CPU 111 makes a decision regarding the presentation of information about this coffee shop S to the user based on the schedule information, which is information about the user's schedule. Here, coffee shop S is an entity that is not explicitly stated in the user's schedule information, or in other words, coffee shop S is an entity that is not included in the user's schedule information.

[0298] In this embodiment, in this case, the CPU 111 determines the content of the information to be presented to the user as information about the coffee shop S based on the user's schedule information.

[0299] Specifically, for example, when the user is heading to a meeting, the CPU 111 causes the terminal device 200 to display information about the coffee shop S, such as "How about a cup of coffee to wake you up?" Furthermore, the CPU 111 causes the terminal device 200 to display, as information about the coffee shop S, for example, information such as "We have products exclusive to this area!" when the user is traveling. When this information is displayed on the terminal device 200, the information may be controlled so as not to overlap with the coffee shop S that is visible to the user.

[0300] Additionally, when the user's schedule information satisfies a predetermined condition, the CPU 111 controls the information about the coffee shop S to be presented to the user so that campaign information about the coffee shop S is included. For example, if the user's available time, as determined based on the user's schedule information, is greater than a predetermined threshold, the CPU 111 controls the information about the coffee shop S presented to the user to include campaign information about the coffee shop S, such as "10% off coffee."

[0301] Here, "campaign information" includes not only coupons but also information about discount codes, sale information, free gifts (for example, novelty gifts or one complimentary drink), and the like.

[0302] Additionally, the CPU 111 controls the information about the coffee shop S that is presented to users who are likely to purchase takeout products, such as users who have upcoming plans, so that campaign information such as "10% off coffee" is included. Users who are likely to order takeout will not reduce the turnover of seats in the restaurant, so when these users visit the restaurant, they contribute to sales. For users who do not have any immediate plans, only the store's busy status and store rating may be presented, and no discount coupons may be presented.

[0303] In this embodiment, the CPU 111 controls the information about the coffee shop S to include information about the campaign of the coffee shop S when the schedule information satisfies a predetermined condition. Furthermore, if the schedule information does not satisfy a predetermined condition, CPU 111 controls the information about coffee shop S so that information about the coffee shop S's campaign is not included. Here, campaign information refers to information that is beneficial to the user. Examples of campaign information include coupons and special price information.

[0304] In this embodiment, information about the user's current schedule and upcoming schedule can be obtained based on the user's schedule information, thereby identifying information that is likely to appeal to the user. In addition, in this embodiment, information about the location and start time of the user's next schedule can be obtained based on the user's schedule information. Here, for example, if the CPU 111 analyzes the user's schedule information and determines that the user is likely to be with people involved in the current schedule or the next schedule, it controls the information to include campaign information about coffee shop S, such as the above-mentioned "10% off coffee." In this case, the possibility that the product will be purchased not only for the user but also for the related parties is increased.

[0305] Furthermore, for example, if the object in the user's vicinity is a hair salon, a decision can be made regarding the presentation of information about the hair salon to the user based on the user's schedule information. Specifically, for example, when referring to a user's schedule information, if the user's last scheduled haircut appointment date was three months ago, the user will be presented with information to advance the haircut, and campaign information such as "30% off cuts today only" will also be included.

[0306] Also, for example, when referencing a user's schedule information, if the user's last scheduled haircut appointment was three weeks ago, information recommending a treatment will be presented to the user, and campaign information such as "30% off treatment today only" will be included.

[0307] Additionally, the information about the coffee shop S or the hair salon presented to the user may be information that encourages the user to make a reservation for a service provided at the coffee shop S or the hair salon.

[0308] Additionally, the CPU 111 may acquire information about the schedules of each of a plurality of users who are present in the vicinity of a certain facility as the schedule information. Then, the CPU 111 may determine, based on the acquired plurality of schedule information, a user to whom the facility information is to be presented from among the plurality of users. In this case, information about the facility is not presented to all of the multiple users who are in the vicinity of the facility, but is presented only to users who satisfy a specific condition, for example.

[0309] An example of a specific condition is the possibility of visiting the facility. In this case, information about the facility is presented only to users whose possibility of visiting the facility exceeds a threshold. This possibility can be calculated, for example, based on the time remaining until the start time of each user's next schedule. If the remaining time is large, it can be determined that the possibility of the user coming to the facility is high, and if the remaining time is small, it can be determined that the possibility of the user coming to the facility is low.

[0310] FIG. 23 is a diagram showing another example of display on the terminal device 200. In FIG. This display example illustrates a case in which a user who has come into view of coffee shop S is presented with information about another coffee shop D that is not in the user's field of view (or is in the user's field of view but is not close enough for the user to recognize).

[0311] The display example shown in FIG. 23 shows a display example when coffee shop S is within the imaging range of camera 214 (not shown in FIG. 23) of terminal device 200 owned by the user. In this example display, in addition to information about coffee shop S, information about coffee shop D, which is not within the imaging range, is also displayed. Here, coffee shop S and coffee shop D are operated by different people but are in the same business. In other words, coffee shop D can be said to be an associated entity that has a predetermined relationship with coffee shop S.

[0312] In this processing example, there are currently no seats available at coffee shop S, and there is a long line of customers waiting to pay at coffee shop S. The time it will take for the user to purchase food and drinks at coffee shop S is estimated to be approximately 15 minutes based on the number of people waiting in line. In this processing example, a message is displayed indicating that the waiting time at coffee shop S is 15 minutes. In addition, in this processing example, information about coffee shop D is displayed, indicating that coffee shop D is 50 m away from the current location, and that the waiting time at coffee shop D is 3 minutes.

[0313] According to the information (an example of schedule information) registered in the calendar of this user who has come into view of coffee shop S, the user's destination (the location of the next appointment) is a concert hall (not shown) at the end of the road in front of the user. Furthermore, coffee shop D presented to the user is located between the current location and the concert hall (an example of the planned venue).

[0314] Coffee shop D is located in such a way that even if the user stops by this coffee shop D, the user will be in time for the start of the concert. In addition, information about Coffee Shop D may be presented to the user if it is determined that the user will be able to make it to the start of the concert on time, based on both the travel time and the waiting time, taking into consideration not only the user's travel time but also the waiting time at Coffee Shop D.

[0315] In this processing example, the CPU 111 first identifies that a coffee shop S exists in the vicinity of the user. Then, based on this user's schedule information, CPU 111 makes a decision regarding presenting to the user information about coffee shop D that is located in a different location from coffee shop S that is located in the user's vicinity. More specifically, the decision is made regarding presenting to the user information about coffee shop D, which is an associated entity that has a predetermined relationship with coffee shop S that is located in the user's vicinity. Here, coffee shop D is not explicitly stated in the user's schedule information.

[0316] In this embodiment, affiliated facilities that are affiliated with coffee shop S are identified using a database described below. That is, in this embodiment, after identifying a facility within the user's field of view, a database is referenced based on this facility to identify related facilities that have a predetermined relationship with that facility.

[0317] Here, it is assumed that there may be a case where a single facility has multiple related facilities in the database. If there are multiple related facilities for a surrounding facility identified as being in the user's vicinity, CPU 111 determines from among the multiple related facilities which related facility to present information about to the user, for example, based on the status information of the surrounding facilities. Specifically, for example, when coffee shop S identified as being in the user's vicinity is crowded as described above, CPU 111 determines the related facility about which information is to be presented to the user based on the congestion status of each of the multiple related facilities. For example, from among the multiple related facilities, a facility that is less crowded than coffee shop S may be selected, or from among the multiple related facilities, the least crowded facility may be selected.

[0318] In this embodiment, when there are multiple associated facilities for coffee shop S, CPU 111 determines, based on the status information of each of the multiple associated facilities, which associated facility should be provided with information about the associated facility to the user. Additionally, if there are multiple related facilities for coffee shop S, the related facility for which information should be presented to the user may be determined from among these multiple related facilities based on the user's level of interest in each of the multiple related facilities.

[0319] Additionally, the CPU 111 may make a decision regarding the presentation of information about facilities related to the coffee shop S to the user based on the situation information about the coffee shop S located in the vicinity of the user.

[0320] Specifically, the CPU 111 may perform control such that, for example, when the degree of congestion at the coffee shop S is low, the presentation of information about related facilities is stopped, and when the degree of congestion at the coffee shop S is high, the presentation of information about related facilities is performed. Furthermore, the CPU 111 may increase or decrease the amount of information about related facilities according to, for example, the degree of congestion of the coffee shop S. Specifically, the greater the degree of congestion of the coffee shop S, the more information about related facilities may be provided.

[0321] Furthermore, one or more of the following conditions may be used to determine whether to present to the user information about coffee shop D, which is an associated facility of coffee shop S located in the user's vicinity. (1) The user prefers coffee shop D to coffee shop S. (2) When the user has been looking at coffee shop S for a certain period of time or more, the user is showing interest in the coffee shop itself. (3) Coffee Shop D has some advantages over Coffee Shop S. Specific examples include when coffee shop D can provide service in a shorter time than coffee shop S, when coffee shop D is cheaper than coffee shop S, or when coffee shop D is having a special sale. (4) Coffee Shop D has entered into a contract and is configured to present information about Coffee Shop D even when the user is not in the vicinity of Coffee Shop D. (5) The user has set up a preference to receive information about Coffee Shop D. (6) Considering the location and start time of the next appointment, it is not possible to stop by Coffee Shop S, but it is possible to stop by Coffee Shop D.

[0322] In addition, presenting information about aroma shop U (see Figure 20) to user Y who is in the vicinity of aroma shop K described above can also be seen as an example of presenting related items that have a predetermined relationship with items in the user's vicinity. Whether or not a user wishes to be presented with related items can be determined based on information such as whether or not the user has enabled the function for presenting related item information, and whether or not the user has paid a fee to have related item information presented.

[0323] Furthermore, one or more of the following conditions may be used to determine whether or not to present to the user information about coffee shop D, which is an associated facility of coffee shop S located in the user's vicinity. (1) The user dislikes Coffee Shop D. (2) Users are not interested in coffee shops. (3) The user has disabled the function to provide related information. (4) Coffee shop D's conditions are worse than those of coffee shop S. Examples of situations in which coffee shop D's conditions are worse than coffee shop S's conditions include when coffee shop D takes longer to provide service than coffee shop S, when coffee shop D's prices are higher than coffee shop S's, or when coffee shop S is having a sale. (5) The operator of information processing system 1 and coffee shop S have entered into a contract to prevent information about coffee shop D from being displayed while the user is in the vicinity of coffee shop S, and the system is set up in this manner. Specifically, for example, a company may have a contract that states that it will not provide information about other stores in the same industry to users who pass by its own coffee shop S, and the contract is set up in this way.

[0324] Here, in order to directly post advertisements in existing stores or to have advertisements displayed on signage in existing stores, it is often necessary to obtain permission from the stores and pay advertising fees. However, when information is displayed on the terminal device 200, which is a personal property of the user, as in this embodiment, information about competing stores of the store that the user is viewing may also be displayed. Also, depending on how the information is displayed, it may be difficult to see the exterior of the store. In this case, there is a risk of disrupting business and damaging the landscape.

[0325] Therefore, when displaying information about related objects that are related to the surrounding objects that the user is viewing, it is preferable to control the display so that the information about the related objects does not overlap as much as possible with the surrounding objects that the user can see. In other words, it is preferable that information about coffee shop D is presented to the user in a way that does not overlap with coffee shop S that is visible to the user.

[0326] Furthermore, it is preferable to control the display of information about coffee shop S so that it is superimposed on coffee shop S, and information about coffee shop D avoids coffee shop S and does not overlap as much as possible with coffee shop S that is visible to the user. The information about coffee shop S may also be displayed so as not to overlap with coffee shop S. In this case, CPU 111 recognizes coffee shop S and controls the display so that the information about coffee shop S does not overlap with coffee shop S visible to the user as much as possible.

[0327] FIG. 24-1 is a diagram showing another example of display on the terminal device 200. In FIG. As an example of a process for displaying information about coffee shop D so as to avoid overlapping as much as possible with coffee shop S that is visible to the user, the information about coffee shop D may be displayed on the edge or corner of the display screen of terminal device 200, as shown by reference numeral 24A in FIG. 24-1. As another example, the CPU 111 may recognize the coffee shop S and display information about the coffee shop D so as to avoid the coffee shop S. As another example, CPU 111 may recognize an area where there are no other facilities, and display information about coffee shop D so that the information about coffee shop D is superimposed on that area. More specifically, the sky or road may be recognized, and information about coffee shop D may be displayed so that the information about coffee shop D is superimposed on the recognized sky or road, as shown by reference numerals 24B and 24C in FIG. 24-2. It is preferable that CPU 111 recognizes road signs, pedestrians, bicycles, cars, etc. around the user and controls the information of coffee shop D so that it does not overlap with these. As one example, if a car approaches the user while information of coffee shop D is being displayed, the information of coffee shop D may be hidden, and the information of coffee shop D may be redisplayed once the car has passed. As another example, the information of coffee shop D may be moved to avoid the car as it moves. Furthermore, CPU 111 may adjust the information about coffee shop D to fit the size of the area allowed to display the information about coffee shop D. For example, it may adjust the position at which a line break occurs for the information about coffee shop D, adjust the character size of the information about coffee shop D, adjust the display direction of the character string of the information about coffee shop D (for example, if the character string is normally displayed horizontally, it may be adjusted to display it vertically or diagonally), or simplify the information about coffee shop D (for example, it may extract part of the information about coffee shop D, express the information about coffee shop D in words that fit the area size, or replace some or all of the character string of the information about coffee shop D with an abbreviation, symbol, emoticon, icon, or the like). In addition, even when the information about coffee shop S identified as being in the user's vicinity is not displayed superimposed on the coffee shop S visible to the user, the same processing as that described above for the case where the information about coffee shop D is not displayed superimposed on the coffee shop S visible to the user may be performed. That is, as an example of a process for displaying information about coffee shop S so as to avoid overlapping as much as possible with the coffee shop S visible to the user, the information about coffee shop S may be displayed on the edge or corner of the display screen of terminal device 200. As another example, the CPU 111 may recognize the coffee shop S and display information about the coffee shop S so as to avoid the coffee shop S. As another example, the CPU 111 may recognize an area where there are no other facilities, and display information about the coffee shop S so that the information about the coffee shop S is superimposed on that area. Furthermore, even when preventing information about coffee shop S from being displayed over coffee shop S that is visible to the user, it is preferable to recognize road signs, pedestrians, bicycles, cars, etc., and control the display so that the information about coffee shop S does not overlap these. In this case, similar to the above, if a car approaches the user while information about coffee shop S is being displayed, the information about coffee shop S may be hidden, and the information about coffee shop S may be redisplayed once the car has passed. As another example, the information about coffee shop S may be moved to avoid the car as it moves. Furthermore, when preventing information about coffee shop S from being displayed superimposed on coffee shop S, the information about coffee shop S may be adjusted to fit the size of the area allowed for displaying the information about coffee shop S, similar to the processing performed when displaying information about coffee shop D.

[0328] The CPU 111 may also acquire information about the schedules of each of a plurality of users who are present around the coffee shop S as the schedule information. Then, the CPU 111 may determine, from among the plurality of users, a user to whom information about the coffee shop D is to be presented, based on the plurality of pieces of schedule information that are acquired. In this case, information about coffee shop D is not presented to all of the multiple users who are in the vicinity of coffee shop S, but is presented only to users who satisfy a specific condition, for example.

[0329] An example of the specific condition is the possibility of visiting coffee shop D. When inviting a user, the possibility of visiting coffee shop D can be set, and in this case, coffee shop D will be presented only to users whose possibility of visiting coffee shop D exceeds a threshold. This possibility can be calculated based on, for example, the remaining time until the user's next appointment, or based on the user's level of interest in coffee shop D.

[0330] In the display examples shown in FIGS. 23, 24-1, and 24-2, the distance to coffee shop D and the waiting time at coffee shop D are displayed as information about coffee shop D, but the displayed information is not limited to this. For example, in addition to this, the user may be presented with information encouraging the user to make a reservation for a service, such as "Coffee shop D is 50 meters away. If you order now, you can pick it up quickly," or "Coffee shop D is 50 meters away. Would you like to order so that it will be ready when you pass by?"

[0331] In other words, the information about coffee shop D presented to the user may be information that encourages the user to make a reservation for a service provided at coffee shop D. Here, the manner in which information encouraging a reservation is displayed includes a manner in which only a reservation button or only a link to a reservation site is displayed. The user who is prompted to make a reservation for the service can make a reservation at coffee shop D, for example, by performing an operation on an application for coffee shop D that is installed on terminal device 200. Furthermore, as described above, when a reservation button or a link to a reservation site is displayed, the user can make a reservation at coffee shop D by operating this reservation button or the link to the reservation site.

[0332] Alternatively, for example, as shown in FIG. 25 (a diagram showing another example of display on the terminal device 200), information on coffee shop S may not be displayed, and only information on coffee shop D may be displayed. In the example shown in FIG. 25, CPU 111 recognizes that coffee shop S is located in the vicinity of the user, and displays information about coffee shop D, which has been identified as a related entity, but does not display information about coffee shop S.

[0333] In the examples shown in Figures 22, 23, 24 and 24-1, information about coffee shop S is also displayed, but coffee shop S is close to the user and visible to the user. In this case, even if information about this coffee shop S is not presented, the user can recognize the name and congestion status of the coffee shop S from the coffee shop S itself. As shown in FIG. 25, if information about objects in the user's vicinity is not presented to the user, and only information about objects related to the objects in the vicinity is presented, the amount of information on the display screen is reduced, making it easier for the user to view the display screen.

[0334] 26 is a diagram showing a display example in which a user who has time to spare until his or her next appointment that is equal to or greater than a predetermined threshold has walked to coffee shop S. In other words, at this time, coffee shop S is a nearby object of the user. In this example, the user who has walked to coffee shop S is a user who likes coffee shop S. Also, in this example, coffee shop S is crowded. In this example, information about a watch shop C is displayed as related entity information about a related entity that has a predetermined relationship with the coffee shop S.

[0335] In this example, a message is displayed informing the user that if the user orders a product sold at coffee shop S, he or she will receive a discount coupon that can be used at watch shop C. More specifically, this example display indicates that if a user orders an item sold at coffee shop S online, they will receive a 5% off coupon that can be used at watch shop C.

[0336] In this example, the CPU 111 displays information about a campaign at a watch shop C as information about an associated entity that has a predetermined relationship with a coffee shop S that has been recognized as an entity in the user's vicinity. More specifically, in this display example, CPU 111 displays information about a campaign at watch shop C, which is valid only if the user uses coffee shop S, which is a nearby property of the user, as information about properties related to coffee shop S.

[0337] In this processing example, the user's waiting time at coffee shop S is 15 minutes. Therefore, even if the user orders an item sold at coffee shop S online, the user will not be able to receive the item until 15 minutes later or later, which leaves the user with some free time. In this case, if the user receives a 5% off coupon that can be used at watch store C as described above, the user is more likely to use this free time to go to watch store C.

[0338] Here, when there are multiple candidates for related objects of a nearby object (here, coffee shop S), it is preferable to select the best candidate from among those candidates. Specifically, it is preferable to select related objects for which information should be presented to the user based on, for example, the waiting time at coffee shop S, the distance from coffee shop S to each candidate, the travel time from coffee shop S to each candidate, the time it takes to receive service at each candidate, the user's interests, or a combination of two or more of these. In this example, the time required to travel between coffee shop S and watch shop C is 2.5 minutes, and the average stay time at watch shop C is 9.5 minutes. In this case, if the user returns to coffee shop S after spending an average amount of time at watch shop C, the user can pick up the product he or she ordered at coffee shop S at just the right time.

[0339] FIG. 27 shows another example of display on the terminal device 200. This display example shows a case where a user whose next appointment is a "meeting" and who has time to spare equal to or greater than a predetermined threshold has moved in front of coffee shop S. In other words, coffee shop S is a nearby object for the user at this time. In this display example, information about a booth that is affiliated with coffee shop S is displayed as information about related objects that have a predetermined relationship with coffee shop S, which has been recognized as an object in the user's vicinity. The booth is located a five-minute walk from coffee shop S. In this example, information about the booth is displayed, including the fact that it takes five minutes to get to the booth, and a link to make a reservation for the booth. In this example display, when the user selects this link, the display screen changes to one for reserving a booth.

[0340] In this display example, the CPU 111 also displays information about a campaign at coffee shop S that is valid only if the booth is used. Specifically, in this example, campaign information is displayed that if you make a reservation for a booth, you will be able to customize drinks sold at coffee shop S for free.

[0341] 27, the information displayed is about a campaign that offers free drink customization at coffee shop S as information about the user's surroundings. Also, the information displayed is about related items to the surroundings, such as the estimated travel time to the booth being five minutes, and a link to make a reservation for the booth.

[0342] FIG. 28 is a flowchart showing the flow of the processing described with reference to FIGS. In this embodiment, when the movement of the user is detected (step S601), information about the next schedule of this user is obtained (step S602). Next, the surrounding objects around the user are sequentially identified (step S603). In other words, the facilities included in the video captured by the camera 214 provided in the terminal device 200 of the user are sequentially identified. Furthermore, in this embodiment, every time a peripheral object is identified, a related object associated with the identified peripheral object is identified (step S604).

[0343] Then, it is determined whether the related object identified in step S604 is within a range that the user can visit before the start time of the next plan (step S605). If it is determined that the related object is within the range where the user can stop by, it is determined whether the related object is on the route the user will take when heading to the location of the user's next appointment (step S606). Furthermore, in this embodiment, it is determined whether the related object has any advantages over the surrounding objects (step S607).

[0344] Then, if it is determined in step S607 that the related object has advantages over the surrounding objects, it is determined that information about the related object is to be displayed (step S608). On the other hand, if it is not determined in step S605 that the related object is within the approaching range, if it is not determined in step S606 that the related object is on the route, or if it is not determined in step S607 that the related object has an advantage over the surrounding objects, the process proceeds to step S609.

[0345] In the process of step S609, it is determined that information about related objects is not to be displayed. The processes of steps S605 to S607 are not essential, and one or two of these processes may be performed. Alternatively, these processes may not be performed, and once a related object is identified in step 604, information about the identified related object may be displayed. Alternatively, information about both the related object and the surrounding object may be displayed.

[0346] 29 is a diagram showing a database that is referenced by the CPU 111 when displaying surrounding object information and related object information. This database is stored in the information storage device 102 of the external server 100 (see FIG. 2), for example. In this embodiment, as shown in FIG. 29, information about facilities is registered in a database. Specifically, in this embodiment, the following information is registered: facility ID, facility name, location information, industry type, rating, current congestion status, estimated stay time, availability of takeout, availability of online reservations / URL for online reservations, whether the facility wishes to be displayed as an affiliated facility, affiliated facilities, and whether or not there is a campaign. The location information may include, for example, longitude and latitude information and coordinate information on an electronic map. This database may also store campaign details, announcements from the facility, and facility opening hours. This database can be considered an example of a database that stores information to be presented to users, or an example of a database that stores information used to determine the information to be presented to users.

[0347] In this embodiment, in step S604, this database is used to identify facilities that have a predetermined relationship with coffee shop S. As an example, the affiliated facilities column of the database is referenced to identify facilities that are affiliated with coffee shop S. Note that facilities in a predetermined relationship are not limited to facilities in an affiliated relationship. As another example, location information may be referenced to identify facilities within a specific distance as facilities in a predetermined relationship. As another example, an industry column may be referenced to identify facilities in the same industry as facilities in a predetermined relationship. Note that facilities that satisfy multiple relationships may be identified, such as facilities in the same industry that are also affiliated. Furthermore, multiple relationships may be prioritized, and if there is a facility that satisfies a first relationship, that facility may be identified, and if there is not, a facility that satisfies a second relationship may be identified.

[0348] In this embodiment, in step S605, as an example, the location information field of the related facility identified in step S604 is referenced, and based on that location information, the current time, the user's current location, and the location and start time of the user's next event, it is determined whether the related facility identified in step S604 is within a distance that can be reached before the start time of the next event. More preferably, in addition to the location information of the related facility, the congestion status field is also referenced, and, for example, the user's possible waiting time at the related facility is taken into consideration when determining whether there is time to receive service at the related facility. In this embodiment, in step S606, as an example, the location information field of the related facility identified in step S604 is referenced, and based on the user's current location and the location and start time of the user's next event, it is determined whether the related facility identified in step S604 is on the route to the location of the next event. In this embodiment, in step S607, as an example, the congestion status column of the surrounding facility identified in step S603 and the congestion status column of the related facility identified in step S604 are referenced, and if the waiting time at the related facility is shorter, it is determined that the related facility has an advantage over the surrounding facility. In this embodiment, as a result of these processes, it is determined that information about coffee shop D should be displayed. In the examples shown in Figures 23, 24-1, 24-2, and 25, information about coffee shop D, which has a predetermined relationship with coffee shop S, is displayed on terminal device 200 owned by a user who is near coffee shop S. Information about the distance to coffee shop D and the waiting time at coffee shop D can be obtained by referring to the database in Figure 29.

[0349] The above-described configurations are not limited to the above-described embodiments and their modifications, and can be modified within the scope of the spirit of the invention. In other words, it is understood that various modifications of the form and details are possible without departing from the spirit and scope of the claims. For example, some of the components described above may be omitted, or other functions may be added to the components described above. Furthermore, although multiple embodiments have been described above, the configuration included in one embodiment may be interchanged with the configuration included in another embodiment, or the configuration included in one embodiment may be added to another embodiment. [Explanation of symbols]

[0350] 1...information processing system, 100...external server, 111...CPU, 200...terminal device, 214...camera, 217...display

Claims

1. An information processing device having a processor, The processor: Identifying the presence of a first object in the vicinity of the user based on an image showing the situation in the vicinity of the user, or based on an electronic map, the user's current location, and the direction in which the user is facing; making a decision regarding the presentation of first object information, which is information about the first object whose existence has been identified, to the user based on schedule information, which is information about the user's schedule; the first object is an object not included in the schedule information; Information processing device.

2. The processor: The information processing device according to claim 1 , further comprising: a determination as to whether or not to present the first object information to the user, and / or a determination as to the content of the first object information to be presented to the user, based on the schedule information.

3. The processor: The information processing apparatus according to claim 1 , wherein when the schedule information satisfies a predetermined condition, the first item information is controlled to include campaign information for the first item.

4. An information processing device having a processor, The processor: Identifying the presence of a second object in the vicinity of the user based on an image showing the situation in the vicinity of the user or an electronic map, the user's current location, and the direction in which the user is facing; Based on schedule information that is information about the user's schedule, a decision is made regarding the presentation of first object information to the user regarding a first object that has a predetermined relationship with the second object whose existence has been identified and that exists in a location different from the second object; the first object is an object not included in the schedule information; Information processing device.

5. The processor: If the schedule information satisfies a predetermined condition, control is performed so that the first item information includes information about a campaign for the first item; If the schedule information does not satisfy a predetermined condition, control is performed so that the first item information does not include information about a campaign for the first item. The information processing device according to claim 1 or 4.

6. The processor: The information processing device according to claim 4 , wherein when second object information about the second object present within the user's field of view is presented to the user, the first object information is also presented to the user.

7. The processor: The information processing device according to claim 1 , wherein the first object information is controlled to be presented to the user so as not to overlap the first object visible to the user.

8. The processor: The information processing device according to claim 4 , wherein the first object information is controlled to be presented to the user so as not to overlap the second object visible to the user.

9. The processor: making a decision regarding presentation of first object information about the first object to the user; The information processing device according to claim 4 .

10. The processor: and controlling the display of information about a campaign for the second item, which becomes effective on the condition that the first item is used, to the user as the second item information. The information processing device according to claim 6 .

11. the second object information presented to the user prompts the user to make a reservation for a service provided by the second object; The information processing device according to claim 6 .

12. The processor: The information processing device according to claim 4 , wherein control is performed so that information on a campaign for the first item, which becomes valid on the condition that the second item is used, is presented as the first item information.

13. The information processing apparatus according to claim 1 , wherein the first object information presented to the user prompts the user to make a reservation for a service provided by the first object.

14. The processor: The information processing apparatus according to claim 4 , wherein, when a plurality of objects are associated with the second object, the first object is determined from the plurality of objects based on status information of the second object.

15. The processor: The information processing apparatus according to claim 4 , wherein, when a plurality of objects are associated with the second object, the first object is determined from the plurality of objects based on status information of each of the plurality of objects.

16. The processor: The information processing apparatus according to claim 4 , wherein, when a plurality of objects are associated with the second object, the first object is determined from the plurality of objects based on the user's level of interest in each of the plurality of objects.

17. The processor: acquiring, as the schedule information, information about a location and a start time of the next schedule of the user who passed by the first object; determining a benefit or a disadvantage for the provider of the first object when the user uses the first object based on the location and start time of the next schedule acquired as the schedule information and the start time of the next reservation of the first object, and making a decision regarding the presentation of the first object information to the user based on the determined benefit or disadvantage; The information processing device according to claim 1 .

18. The processor: The information processing apparatus according to claim 17 , wherein a predicted usage time of the first object by the user is determined as the benefit or disadvantage.

19. The schedule information includes a type of the user's schedule, The processor: making a decision regarding presentation of the first object information to the user based on the type; The information processing device according to claim 1 or 4.

20. the schedule information includes information about people related to the user's schedule; The processor: making a decision regarding presentation of the first object information to the user based on the number of people; The information processing device according to claim 1 or 4.

21. The processor: As the schedule information, at least the location and start time of the user's schedule are acquired; The information processing device according to claim 1 or 4, wherein a decision is made regarding the presentation of the first object information to the user based on the remaining time from the current time to the start time and the required time, which is the time required to travel from the current location to the event location.

22. The processor: The information processing apparatus according to claim 1 , further comprising: a determination as to whether the first object information is to be presented to the user based on the acquired schedule information and status information of the first object.

23. The processor: The information processing apparatus according to claim 4 , wherein a decision is made regarding presentation of the first object information to the user based on the acquired schedule information and status information of the second object.

24. The information processing device according to claim 1 , wherein the information processing device identifies objects present around the user based on an image captured by a camera that moves together with the user.

25. the information processing device is a mobile device that moves with the user, The processor: The information processing device according to claim 1 , wherein objects present in the vicinity of the user are identified based on a current location of the user and a direction in which the user is facing.

26. The information processing device is a device worn in front of the user's eyeballs. The information processing device according to claim 1 or 4.

27. the information processing device has a see-through display, the processor associates the first object information with a real object in real space visually recognized by the user via the display; 27. The information processing device according to claim 26.

28. The processor: acquiring information about the schedule of each of a plurality of users present in the vicinity of the first object as the schedule information; Identifying a user who satisfies a specific condition from among the plurality of users based on the schedule information, and determining the identified user as a user to whom the first object information will be presented. The information processing device according to claim 1 .

29. The processor: acquiring information about a schedule of each of the plurality of users present in the vicinity of the second object as the schedule information; Identifying a user who satisfies a specific condition from among the plurality of users based on the schedule information, and determining the identified user as a user to whom the first object information will be presented. The information processing device according to claim 4 .

30. the information processing device communicates with a mobile device that moves with the user; The processor: The information processing device according to claim 1 or 4, wherein the first object information is controlled to be displayed on a display of the mobile device.

31. A function for identifying the presence of a first object in the vicinity of a user based on an image showing the situation around the user, or based on an electronic map, the user's current location, and the direction the user is facing; a function of making a decision regarding the presentation of first object information, which is information about the first object whose existence has been identified, to the user based on schedule information, which is information about the user's schedule; It is a program to make a computer realize the first object is an object not included in the schedule information; program.

32. A function for identifying the presence of a second object in the vicinity of a user based on an image showing the situation around the user or an electronic map, the user's current location, and the direction the user is facing; a function of making a decision on the presentation of first object information to the user, the first object having a predetermined relationship with the second object whose existence has been identified and existing in a location different from the second object, based on schedule information that is information about the user's schedule; It is a program to make a computer realize the first object is an object not included in the schedule information; program.

Images