Information processing device

Abstract

The present invention is an information provision server, wherein an identification unit identifies types of objects present in a user's field of view, and outputs type information indicating the identified types. A selection unit selects message information from among a plurality of pieces of message information on the basis of the type information and related information which is related to the objects. A supply unit supplies one piece of the message information to a wearable device. The related information includes: at least one piece of information from among environment information relating to the environment at the location where the object is present and time information indicating the time at which the identification unit identified the type of the object; and location information indicating the location where the object is present.

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] XR glasses that apply XR technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) have become widespread. For example, Patent Document 1 below discloses an AR display management system including a portable terminal device and an authentication device that performs display authentication of AR images. The portable terminal device acquires location information indicating the location of the device and directional information indicating the imaging direction of an imaging unit. The portable terminal device transmits an AR image display authentication request message to the authentication device. The display authentication request message includes an AR image identifier, location information, and directional information. The authentication device authenticates whether to allow display of the AR image corresponding to the identifier based on the identifier, location information, and directional information included in the received display authentication request message, and transmits a response message including the authentication result to the portable terminal device. The portable terminal device determines whether the AR image can be displayed based on the received response message, and if it determines that the AR image can be displayed, displays the AR image superimposed on the captured image displayed on the display unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-87091 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, when multiple AR images are virtually placed in the real space around the user, the user is forced to view multiple AR images. This results in a problem of reducing the display area of ​​the real space that the user can see through the XR glasses. In addition, the increased amount of unnecessary information can distract the user.

[0005] An object of the present invention is to provide useful information to users. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention includes an identification unit that identifies the type of object present in a user's field of vision and outputs type information indicating the identified type; a selection unit that selects one piece of message information from among multiple pieces of message information based on the type information and related information related to the object; and a supply unit that supplies the selected piece of message information to an output device, wherein the related information includes at least one of environmental information regarding the environment at the location where the object is located and time information indicating the time when the identification unit identified the type of the object, and location information indicating the location where the object is located. [Effects of the Invention]

[0007] According to one aspect of the present invention, useful information is provided to the user. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the overall configuration of an information processing system 1 according to an embodiment. [Figure 2] 1 is an explanatory diagram showing the appearance of AR glasses 30. FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of the AR glasses 30. [Figure 4] FIG. 2 is a block diagram showing the configuration of a terminal device 40. [Figure 5] 2 is a block diagram showing the configuration of an information providing server 20. FIG. [Figure 6]FIG. 2 is an explanatory diagram showing an example of an object identification database DB1. [Figure 7] FIG. 2 is an explanatory diagram showing an example of a message information database DB2. [Figure 8] 10 is a flowchart showing the operation of the processing device 206. [Figure 9] FIG. 2 is an explanatory diagram showing an example of a message information database DB2. [Figure 10] 10 is an explanatory diagram schematically showing the relative position between product name B and the user. FIG. 11 is a flowchart showing the operation of a processing device 406. FIG. [Figure 11] 10 is an explanatory diagram showing an example of an output mode of message information M. FIG. [Figure 12] 10 is an explanatory diagram showing an example of an output mode of message information M. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment The configuration of an information processing system 1 including an information processing device according to a first embodiment of the present invention will be described below.

[0010] A-1. System Configuration 1 is a block diagram showing the overall configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes a wearable device 10 and an information providing server 20. The information providing server 20 is an example of an information processing device.

[0011] In this embodiment, the information processing system 1 is a system that uses AR technology to present various types of information to a user wearing AR glasses 30 (described later). Here, AR technology refers to a technology that allows a user wearing a device such as a see-through head-mounted display to visually recognize information indicated by a virtual object by superimposing the virtual object on a coordinate space of the real world. The AR glasses 30 may display virtual objects and also output other types of information, such as audio information. Information provided to a user using the AR glasses 30 is referred to as "message information M."

[0012] The message information M may be, for example, a still image, a video, a 3DCG model, or text. In this embodiment, the message information M is information displayed mainly for the purpose of promoting the sale of a product P sold in a store. The product P is an example of an object. The message information M may be information indicating the type of the product P, for example, image information indicating an explanation of the product name B or audio information indicating an explanation of the product name B.

[0013] The wearable device 10 and the information providing server 20 are connected to each other so that they can communicate with each other via a communication network N. Although only one wearable device 10 is shown in FIG. 1, this number is merely an example, and the information processing system 1 can include any number of wearable devices 10.

[0014] The wearable device 10 is a device worn by a user to perceive message information M. The wearable device 10 includes AR glasses 30 and a terminal device 40. The AR glasses 30 and the terminal device 40 are communicatively connected to each other. The terminal device 40 is communicatively connected to the information providing server 20 to acquire message information M to be output by the AR glasses 30 from the information providing server 20. The terminal device 40 also outputs the message information M acquired from the information providing server 20 from the AR glasses 30. The terminal device 40 is preferably a mobile terminal device such as a smartphone or a tablet.

[0015] A-2.AR Glasses 30 The AR glasses 30 are a see-through head-mounted display worn on the user's head. The AR glasses 30 display virtual objects on display panels provided in each of the binocular lenses 310A, 310B under the control of the terminal device 40. Note that, for example, a goggle-shaped see-through head-mounted display having the same functions as the AR glasses 30 may be used as the display device.

[0016] 2 is an explanatory diagram showing the external appearance of the AR glasses 30. The AR glasses 30 have temples 301 and 302, a bridge 303, frames 304 and 305, rims 306 and 307, lenses 310A and 310B, an imaging lens LEN, and a light transmitting and receiving unit SEN that can be seen from the outside.

[0017] The bridge 303 is provided with an imaging lens LEN constituting the imaging device 324 shown in FIG. 3 and a light transmitting / receiving unit SEN constituting the 3D scanner 325.

[0018] The frame 304 is provided with a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays an image based on control from, for example, a terminal device 40 described later. The optical member for the left eye is an optical member that guides light emitted from the display panel for the left eye to a lens 310A. The frame 304 is also provided with a speaker 322 described later.

[0019] A display panel for the right eye and an optical member for the right eye are provided on the frame 305. The display panel for the right eye displays an image, for example, under control of the terminal device 40. The optical member for the right eye is an optical member that guides light emitted from the display panel for the right eye to the lens 310B. The frame 305 is also provided with a speaker 322, which will be described later.

[0020] Rim 306 holds lens 310A, and rim 307 holds lens 310B.

[0021] Each of lenses 310A and 310B has a half mirror. The half mirror of lens 310A transmits light representing real space, thereby guiding the light representing real space to the user's left eye. The half mirror of lens 310A also reflects light guided by the optical member for the left eye to the user's left eye. The half mirror of lens 310B transmits light representing real space, thereby guiding the light representing real space to the user's right eye. The half mirror of lens 310B also reflects light guided by the optical member for the right eye to the user's right eye.

[0022] When a user wears the AR glasses 30, the lenses 310A and 310B are positioned in front of the user's left and right eyes. The user wearing the AR glasses 30 can visually recognize a real space represented by light transmitted through the lenses 310A and 310B superimposed on an image projected by a projection device 321 (described later).

[0023] 3 is a block diagram showing the configuration of the AR glasses 30. In addition to the above-mentioned temples 301 and 302, bridge 303, frames 304 and 305, rims 306 and 307, lenses 310A and 310B, imaging lens LEN, and light transmitting and receiving unit SEN, the AR glasses 30 include a projection device 321, a speaker 322, a communication device 323, an imaging device 324, a 3D scanner 325, a GPS device 326, a storage device 327, a processing device 328, and a bus 329. The projection device 321 and the speaker 322 are examples of output devices.

[0024] 3 are stored in, for example, frames 304 and 305. The projection device 321, speaker 322, communication device 323, imaging device 324, 3D scanner 325, storage device 327, and processing device 328 are connected to one another by a bus 329 for communicating information. The bus 329 may be configured using a single bus, or may be configured using different buses between each element of the device, etc.

[0025] The projection device 321 includes a lens 310A, a display panel for the left eye, an optical member for the left eye, a lens 310B, a display panel for the right eye, and an optical member for the right eye. The projection device 321 functions as a transmissive display disposed between the user's eyes and an object. The projection device 321 displays an image based on control from the terminal device 40. In this embodiment, the image displayed by the projection device 321 is, for example, image information displayed as message information M.

[0026] The speaker 322 outputs sound to the user. The speaker 322 is located in each of the frames 304 and 305. The speaker 322 may not be located in each of the frames 304 and 305, but may be located in, for example, one of the frames 304 and 305, at least one of the temples 301 and 302, or the bridge 303. The speaker 322 is controlled by the terminal device 40 directly or via the processing device 328 of the AR glasses 30. The speaker 322 outputs, for example, audio information to be output as message information M. The speaker 322 may not be included in the AR glasses 30 and may be separate from the AR glasses 30.

[0027] The communication device 323 communicates with the terminal device 40 using wireless communication or wired communication. In this embodiment, the communication device 323 communicates with the communication device 403 (see FIG. 4) of the terminal device 40 using short-range wireless communication such as Bluetooth (registered trademark).

[0028] The imaging device 324 captures an image of a subject and outputs captured image information indicating the captured image (hereinafter referred to as a "captured image"). In this embodiment, the imaging range of the imaging device 324, for example, coincides with the user's field of view or includes the entire area of ​​the user's field of view. Therefore, the captured image information includes information on the appearance of objects (physical objects) present in the user's field of view. For example, when the user is in a store, images are captured of products lined up on a rack in front of the user or products held by the user.

[0029] The captured image generated by the imaging device 324 is transmitted as captured image information to the terminal device 40 via the communication device 323. The captured image is an example of a first image. The imaging device 324 repeatedly captures images at predetermined imaging intervals, and transmits the generated captured image information to the terminal device 40 each time an image is captured.

[0030] The imaging device 324 has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 2). For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0031] The 3D scanner 325 outputs measured shape information indicating the results of measuring the three-dimensional shape of an object in the user's vicinity. The 3D scanner 325 is an example of a measurement device. The measurement range of the 3D scanner 325, for example, coincides with the user's field of view or includes the entire area of ​​the user's field of view. Therefore, the measured shape information includes information about the three-dimensional shape of the object present in the user's field of view.

[0032] In this embodiment, the 3D scanner 325 irradiates the measurement range with light from a light source provided in the light-transmitting / receiving unit SEN. The light irradiated from the light source is pattern light having a pattern such as a stripe pattern. The 3D scanner 325 also receives reflected light with a light-receiving unit provided in the light-transmitting / receiving unit SEN. The time from light irradiation to reception of the reflected light is proportional to the distance to the object from which the light is reflected. The 3D scanner 325 measures the time from light reception for each point within the measurement range and calculates the distance to each point within the measurement range. Furthermore, the unevenness of the object is recognized due to distortion of the pattern reflected on the surface of the object. In this way, the 3D scanner 325 can obtain the three-dimensional coordinates of the object surface and generate measured shape information indicating the three-dimensional shape of the object.

[0033] The measurement shape information generated by the 3D scanner 325 is transmitted to the terminal device 40 via the communication device 323. The 3D scanner 325 repeats measurements at predetermined measurement intervals, and transmits the generated measurement shape information to the terminal device 40 each time a measurement is taken.

[0034] The shape of the object may be detected from captured images captured by the imaging device 324, rather than being measured by the 3D scanner 325. In this case, the imaging device 324 may be, for example, a stereo camera having multiple cameras, which capture images of the object from different angles. The processing device 328 calculates the three-dimensional shape of the object using the multiple captured images captured from different angles.

[0035] The GPS device 326 receives radio waves from multiple satellites and generates location information from the received radio waves. The location information indicates the location of the AR glasses 30. The location information may be in any format as long as it can identify the location. In this embodiment, the location information indicates, for example, the latitude and longitude of the AR glasses 30. Note that the location information may be obtained using means other than the GPS device 326. For example, information identifying the name of the facility where the AR glasses 30 are located may be received from a beacon or the like installed in the facility. In this embodiment, the location of the AR glasses 30 is the location of the wearable device 10. The GPS device 326 may be installed in the terminal device 40. In this case, the GPS device 326 generates location information indicating the location of the terminal device 40. Furthermore, the location of the wearable device 10 is the location of the terminal device 40.

[0036] The storage device 327 is a recording medium readable by the processing device 328. The storage device 327 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 327 stores a program PG1. The program PG1 is a program for operating the AR glasses 30.

[0037] The processing unit 328 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0038] The processing device 328 reads the program PG1 from the storage device 327. The processing device 328 executes the program PG1 to function as the operation control unit 330. The operation control unit 330 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0039] The operation control unit 330 controls the operation of the AR glasses 30. For example, the operation control unit 330 provides the projection device 321 with a control signal for image display that the communication device 323 has received from the terminal device 40. The projection device 321 displays an image indicated by the control signal for image display. The operation control unit 330 also provides the speaker 322 with a control signal for audio output that the communication device 323 has received from the terminal device 40. The speaker 322 emits a sound indicated by the control signal for audio output. The operation control unit 330 also transmits captured image information captured by the imaging device 324, measured shape information measured by the 3D scanner 325, and location information generated by the GPS device 326 to the terminal device 40.

[0040] A-3. Terminal device 40 4 is a block diagram showing the configuration of terminal device 40. Terminal device 40 includes a touch panel 401, a communication device 403, a storage device 405, a processing device 406, and a bus 407. Touch panel 401, communication device 403, storage device 405, and processing device 406 are interconnected by bus 407 for communicating information. Bus 407 may be configured using a single bus, or may be configured using different buses for each device.

[0041] The touch panel 401 displays various information to the user and detects touch operations by the user. The touch panel 401 serves as both an input device and an output device. For example, the touch panel 401 is configured by bonding a touch sensor unit capable of detecting touch operations between a cover glass and a display panel such as a liquid crystal display panel or an organic EL display panel. For example, when the user's finger is in contact with the touch panel 401, the touch panel 401 periodically detects the contact position of the user's finger on the touch panel 401 and transmits touch information indicating the detected contact position to the processing device 406.

[0042] The communication device 403 communicates with the AR glasses 30 using wireless communication or wired communication. In this embodiment, the communication device 403 communicates with the communication device 323 (see FIG. 3) using short-range wireless communication of the same type as that used by the communication device 323 of the AR glasses 30. The communication device 403 also communicates with the information providing server 20 using wireless communication or wired communication. In this embodiment, the communication device 403 has an interface connectable to the communication network N, and communicates with the communication device 203 (see FIG. 5) of the information providing server 20 via the communication network N.

[0043] The storage device 405 is a recording medium readable by the processing device 406. The storage device 405 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 405 stores a program PG2. The program PG2 is a program for operating the terminal device 40.

[0044] Processing unit 406 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of a processor and a CPU is an example of a computer.

[0045] The processing device 406 reads the program PG2 from the storage device 405. The processing device 406 executes the program PG2 to function as an operation control unit 410. The operation control unit 410 may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA.

[0046] The operation control unit 410 controls the operation of the terminal device 40. In this embodiment, the operation control unit 410 transmits the captured image information, measured shape information, and position information received from the AR glasses 30 to the information providing server 20.

[0047] A-4. Information providing server 20 5 is a block diagram showing the configuration of the information providing server 20. The information providing server 20 includes an input device 201, a display device 202, a communication device 203, a storage device 205, a processing device 206, and a bus 207. The input device 201, the display device 202, the communication device 203, the storage device 205, and the processing device 206 are interconnected by the bus 207 for communicating information. The bus 207 may be configured using a single bus, or may be configured using different buses for each device.

[0048] The input device 201 is a device that accepts operations performed by an administrator of the information processing system 1. For example, the input device 201 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. When the input device 201 includes a touch panel, the input device 201 may also function as the display device 202.

[0049] The display device 202 is a device that displays images. The display device 202 displays various images under the control of the processing device 206. The images may include text information. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used as the display device 202.

[0050] The communication device 203 communicates with the terminal device 40 (see FIGS. 1 and 4) using wireless communication or wired communication. In this embodiment, the communication device 203 has an interface connectable to a communication network N, and communicates with the terminal device 40 via the communication network N.

[0051] The storage device 205 is a recording medium readable by the processing device 206. The storage device 205 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 205 stores a program PG3, an object identification database DB1, and a message information database DB2. Details of these databases will be described later. In FIG. 5, the word "database" is written as "DB." The program PG3 is a program for operating the information providing server 20.

[0052] Processing unit 206 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of a processor and a CPU is an example of a computer.

[0053] The processing device 206 reads the program PG3 from the storage device 205. By executing the program PG3, the processing device 206 functions as an acquisition unit 210, a determination unit 211, a selection unit 212, and a supply unit 213. At least one of the acquisition unit 210, the determination unit 211, the selection unit 212, and the supply unit 213 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA. The acquisition unit 210, the determination unit 211, the selection unit 212, and the supply unit 213 will be described in detail below.

[0054] A-5. Object identification database DB1 Next, the object identification database DB1 will be described. The object identification database DB1 is used to identify the product name B performed by the identification unit 211, which will be described later. The object identification database DB1 is a database that associates at least one of measured shape information and captured image information with the product name B. It is assumed that the object identification database DB1 registers at least a product P whose product name B is registered in the message information database DB2. According to the example of the message information database DB2 in FIG. 7, which will be described later, the object identification database DB1 includes at least information related to "AA soda" and information related to "BB snack."

[0055] Fig. 6 is an explanatory diagram showing an example of the object identification database DB1. The object identification database DB1 has a plurality of records E. The object identification database DB1 shown in Fig. 6 includes three records E1 to E3. One record E associates a product name B with at least one of the reference shape information 502 or the reference image information 504. In one record E, the product name B may be associated with only one of the reference shape information 502 or the reference image information 504.

[0056] Product name B is type information indicating the type of product P. Product P is an example of an object, and product name B is an example of type information indicating the type of object. Specifically, product name B in record E1 is "AA Soda," product name B in record E2 is "BB Snack," and product name B in record E3 is "GG Juice." In this embodiment, the type of product P is identified by product name B. However, without being limited to this, the type of product P may also be identified by a code consisting of an arrangement of alphanumeric characters, for example.

[0057] The reference shape information 502 is information indicating the three-dimensional shape of a reference object defined for each product name B. The reference shape information 502 indicates the results of 3D scanning the reference object using, for example, a 3D scanner. In FIG. 6, the reference shape information 502 indicates the data name of the data output as a result of 3D scanning the reference object using a predetermined method. For example, the reference shape information 502 "aaa.stp" of record E1 is data output as a result of selecting one AA soda bottle produced in a factory as the reference object and 3D scanning it using a 3D scanner.

[0058] Reference image information 504 is an image obtained by capturing an image of the appearance of a reference object determined for each product name B. Reference image information 504 is an example of a second image. In FIG. 6, reference image information 504 indicates the data name of data output as a result of capturing an image of the reference object. Note that reference image information 504 may include multiple images of the reference object captured from different directions. For example, "aaa.jpg", which is reference image information 504 for record E1, is data output as a result of capturing an image of a bottle of AA soda selected from those produced in a factory using an imaging device.

[0059] The identification unit 211, described later, compares the reference shape information 502 with the measured shape information, and the reference image information 504 with the captured image information, to identify the product name B of the product P indicated by the measured shape information and the captured image information. As in this embodiment, identifying an object using the captured image information and the measured shape information improves the accuracy of identifying the type of product P. For example, using only the captured image information may result in the product P appearing on a poster being mistakenly recognized as the actual product P. By using the measured shape information, a flat image such as a poster image is no longer recognized as the product P (object), preventing message information M from being output in an inappropriate situation. Furthermore, for example, if the shape of the product P is not distinctive (e.g., a general-purpose pillow packaging bag), it is difficult to obtain object identification information using only the measured shape information. In this case, the type of product P can be accurately identified by using the captured image information together with the measured shape information. Meanwhile, in the object identification database DB1, there are cases where only either the reference shape information 502 or the reference image information 504 is associated with the product name B. When the reference shape information 502 is associated with the product name B, the identification unit 211 uses the reference shape information 502 and the measured shape information to identify the product name B. When the reference image information 504 is associated with the product name B, the identification unit 211 uses the reference image information 504 and the captured image information to identify the product name B.

[0060] In the case of a product P that comes in multiple sizes, such as a beverage, a product name B may be set for each size. For example, assume that "AA soda" comes in a 500ml plastic bottle and a 2L plastic bottle. In this case, separate product names B, such as "AA soda 500ml plastic bottle" and "AA soda 2L plastic bottle," may be set for the two products P, or the two products P may be collectively set with the product name B "AA soda."

[0061] Furthermore, for example, one type information (product name B) may be set for multiple products P belonging to the same category. For example, the reference shape information 502 and reference image information 504 for "AA soda" and the reference shape information 502 and reference image information 504 for "GG juice" may be associated with the type information "soft drink."

[0062] A-6. Message information database DB2 Next, details of the message information database DB2 will be described. The message information database DB2 is used for selecting message information M performed by the selection unit 212, which will be described later. The message information database DB2 associates product names B, related information R, and message information M. The product name B stored in the message information database DB2 indicates the type of product P (object) that triggers the output of message information M on the AR glasses 30.

[0063] In general AR technology, a product name B and message information M are associated one-to-one. In contrast, in this embodiment, message information M is associated with a pair of product name B and related information R. The related information R includes at least one of environmental information about the environment at the location where the product P is located and time information indicating the time when the identification unit 211, which will be described later, identified the type of product P, and location information indicating the location where the product P is located. By associating message information M with product name B and related information R, the conditions for outputting message information M can be set in more detail. Therefore, more useful information can be provided to the user.

[0064] In this embodiment, the position where the product P exists is approximated to the position of the wearable device 10 worn by the user who has the product P in their field of view, that is, the position of the AR glasses 30 having the GPS device 326.

[0065] 7 is an explanatory diagram showing an example of the message information database DB2. The message information database DB2 has a plurality of records D. The message information database DB2 shown in FIG. 7 includes six records D1 to D6. One record D associates a product name B, location information 601, output condition information 602, and message information M. The location information 601 and the output condition information 602 correspond to related information R.

[0066] The product name B indicates the type of product P. Specifically, the product name B of records D1 to D4 is "AA Soda," and the product name B of record D5 is "BB Snack."

[0067] Location information 601 identifies the location where product P is located. In the example of FIG. 7, the location information is a facility name. Specifically, the location information 601 of records D1 and D2 is "CC Mart DD Branch," and the location information 601 of records D3 to D5 is "CC Mart EE Branch." "CC Mart DD Branch" and "CC Mart EE Branch" are different branches belonging to the same chain and are located in different places.

[0068] 7, facility names are shown as location information, but the location information is not limited to this and may be, for example, latitude and longitude information.

[0069] The output condition information 602 indicates conditions for outputting the message information M. In the first embodiment, the output condition information 602 is at least one of environmental information related to the environment at the location where the product P is located and time information indicating the time when the identification unit 211, which will be described later, identified the type of the product P. The environmental information may be, for example, weather information. Examples of weather information include sunny, cloudy, etc., whether or not there is precipitation, temperature, humidity, wind direction and speed, etc. If the product P is located inside a building, the temperature may be the room temperature inside the building where the product P is located, or the outside temperature outside the building.

[0070] Furthermore, the time information indicates the time at the location where the product P is located. There may be a time difference between the local time at the location where the information providing server 20 is placed and the local time at the location where the product P is located, but in this embodiment, the time information in the output condition information 602 indicates the local time at the location where the product P is located.

[0071] Specifically, the output condition information 602 of record D1 is time information "time 10:00 to 15:59." The output condition information 602 of record D2 is time information "time 22:00 to 5:59." The output condition information 602 of record D3 is environmental information (weather information) "temperature 30°C or higher." The output condition information 602 of record D4 is environmental information "temperature less than 15°C." The output condition information 602 of record D5 is "other than the above," which indicates a case where no other output condition applies. The output condition information 602 of record D6 is environmental information "temperature less than 15°C."

[0072] 7, the output condition information 602 is configured with one type of condition, but the output condition information 602 may include two or more types of conditions. For example, the output condition information 602 may be a combination of time information such as "time 10:00 to 15:59" and "temperature 30°C or higher" and weather information. Furthermore, the environmental information in the output condition information 602 is not limited to weather information, and may also be, for example, the degree of congestion in the surrounding area.

[0073] The message information M is information output by the AR glasses 30. FIG. 7 shows a case where the message information M is mainly text information. The text information may be, for example, purchase benefits for the product P, nutritional information for the product P, or the like. The message information M may also be video, still images, audio information, or the like. In this case, FIG. 7 shows the file name of the video, still images, or audio information.

[0074] As a specific example, the message information M of record D1 is text information saying "Get 10% off now." The text information is displayed as characters on the AR glasses 30. The message information M of record D2 is "aaa.mp4." Record D2 indicates that video data "aaa.mp4" will be played as message information M. The message information M of record D3 is text information saying "Buy two, get one free." The message information M of record D4 is text information saying "50 yen off when purchased together with BB snacks." The message information M of record D5 is "Ingredients and nutritional information.jpg." Record D5 indicates that a still image including the ingredient names and nutritional information will be displayed as message information M. The message information M of record D6 is text information saying "50 yen off when purchased together with AA soda."

[0075] The message information database DB2 may store setting information for outputting the message information M together with the message information M. For example, when the message information M is text information, the setting information specifies the font type, font size, font color, etc. for displaying the text information. The setting information may also include the display position of the message information M in the field of view of the user wearing the AR glasses 30.

[0076] Various pieces of information may be registered in the message information database DB2 by, for example, a sales representative of the manufacturer of the product P, or by the owner or sales representative of a store that sells the product P. The person in charge of registering information in the message information database DB2 accesses the message information database DB2 via a communication network N from, for example, their own information processing device, and updates the contents of the message information database DB2 as needed. By updating the message information database DB2, the message information M output to the user can be changed in real time. This allows for more effective sales promotion of the product P using, for example, the AR glasses 30.

[0077] A-7. Details of the processing device 206 As described above, the processing device 206 executes the program PG3 to function as the acquisition unit 210, the identification unit 211, the selection unit 212, and the supply unit 213. Each functional unit will be described in detail below.

[0078] The acquisition unit 210 acquires captured image information, measured shape information, and position information from the AR glasses 30. As described above, the captured image information, measured shape information, and position information are transmitted from the AR glasses 30 to the terminal device 40, and further transmitted from the terminal device 40 to the information providing server 20. The acquisition unit 210 acquires the captured image information, measured shape information, and position information received by the communication device 203.

[0079] Furthermore, the acquisition unit 210 acquires weather information for the position of the AR glasses 30, i.e., the position corresponding to the position information acquired from the AR glasses 30, as necessary. The weather information may be acquired, for example, from a weather information providing server (not shown). The weather information providing server acquires, for example, observation values ​​at weather observation stations installed in various locations via the communication network N. The weather information providing server also predicts weather conditions at an arbitrary position based on the observation values ​​at each weather observation station, and provides the weather information for the arbitrary position to the information providing server 20, etc. Furthermore, the acquisition unit 210 may acquire weather information from the AR glasses 30 if the AR glasses 30 are capable of detecting weather information. An example of a case in which the AR glasses 30 are capable of detecting weather information is when the AR glasses 30 are equipped with a thermometer, a hygrometer, or a wind speed and direction meter.

[0080] Furthermore, the acquisition unit 210 acquires, as necessary, the current time, i.e., local time information, at the position of the AR glasses 30, i.e., at the position corresponding to the position information acquired from the AR glasses 30. The acquisition unit 210 may calculate the local time information based on the position information acquired from the AR glasses 30, for example.

[0081] The identification unit 211 identifies product name B, which is the type of product P present in the user's field of view, and outputs product name information indicating the identified type. In this embodiment, the identification unit 211 identifies product name B using the measured shape information and captured image information acquired by the acquisition unit 210. More specifically, the identification unit 211 compares the reference shape information 502 and the measured shape information of each record E in the object identification database DB1, and the reference image information 504 and the captured image information of each record E. If there is a record E in which the degree of match between the reference shape information 502 and the measured shape information is equal to or greater than a first predetermined value and the degree of match between the reference image information 504 and the captured image information is equal to or greater than a second predetermined value, the identification unit 211 identifies the product name B of the record E as the product name B of the product P present in the user's field of view.

[0082] Hereinafter, the product name B detected by the identification unit 211 will be referred to as a specific product name BX. If the identification unit 211 cannot detect the product name B of the product P in the user's field of view, it outputs information indicating "no corresponding product."

[0083] That is, the identification unit 211 identifies the product name B of the product P based on a first comparison result obtained by comparing measured shape information indicating the results of measuring the three-dimensional shape of the product P with reference shape information 502 indicating the three-dimensional shape of a reference object defined for each product name B. Furthermore, the identification unit 211 identifies the product name B based on a second comparison result obtained by comparing captured image information obtained by capturing an image of the product P with reference image information 504 obtained by capturing an image of the appearance of the reference object. The identification unit 211 may identify the product name B using both the first comparison result and the second comparison result. Furthermore, the identification unit 211 may identify the product name B using either the first comparison result or the second comparison result.

[0084] The identification unit 211 may identify the product name B using AI (Artificial Intelligence). In this case, a trained model for the product P is stored in the storage device 205, for example. The trained model is a trained model that associates at least one of the shape (e.g., a shape identified based on the reference shape information 502) or the appearance (e.g., the appearance identified based on the reference image information 504) of the product P with the product name B, using deep learning that uses, for example, a convolutional neural network. When at least one of measured shape information or captured image information is input as input information, the trained model outputs the product name B as output information.

[0085] The selection unit 212 selects one piece of message information M from the plurality of pieces of message information M based on the specific product name BX and related information related to the product P. More specifically, the selection unit 212 refers to the message information database DB2 to search for a record D that includes the specific product name BX. Taking FIG. 7 as an example, if the specific product name BX is "AA soda," the records D that include "AA soda" are records D1 to D5.

[0086] Next, the selection unit 212 refers to the location information 601 of the record D including the specific product name BX and determines whether there is a record D that matches the location information of the AR glasses 30 or includes the location information of the AR glasses 30. As described above, the location of the AR glasses 30 is approximated by the location of the product P identified based on the specific product name BX.

[0087] 7 as an example, the location information 601 of records D1 to D5 associated with the specific product name BX, "AA Soda," is "CC Mart DD Branch" and "CC Mart EE Branch." If the location of the AR glasses 30 is not "CC Mart DD Branch" or "CC Mart EE Branch," there is no corresponding record, and the selection unit 212 ends the process. In this case, the message information M is not selected, and the message information M is not output to the AR glasses 30.

[0088] If there is a record D that includes location information 601 that matches the location information of the AR glasses 30, the selection unit 212 refers to the output condition information 602 of the record D and determines whether the output condition is met. Whether the output condition is met is determined based on the environmental information acquired by the acquisition unit 210 or time information indicating the time when the identification unit 211 identified the product name B. The selection unit 212 requests the acquisition unit 210 to acquire environmental information as necessary.

[0089] If the output condition is met, the message information M of the record D including the output condition information 602 is identified as the message information M to be provided to the wearable device 10. If the output condition is not met, the message information M to be provided to the wearable device 10 is not identified, and the message information M is not output to the AR glasses 30.

[0090] Using FIG. 7 as an example, if the specified product name BX is "AA Soda" and the AR glasses 30 are located at "CC Mart DD Branch," records D1 and D2 are to be determined. Since the output condition information 602 in records D1 and D2 is a condition related to time, the selection unit 212 determines whether the output condition is met based on the time (which can be considered roughly equivalent to the current time) when the identification unit 211 identified product name B. For example, if the time when the identification unit 211 identified product name B is 10:00, this matches the output condition of record D1. Therefore, message information M ("10% off now") of record D1 is selected. On the other hand, if the time when the identification unit 211 identified product name B is 9:55, this does not match the output condition of either record D1 or D2. Therefore, message information M is not selected. In addition, if there is a time difference between the time at the location of the information providing server 20 and the time at the location where the product P is located, the time at which the identification unit 211 identified the product name B is converted to the local time at the location where the product P is located, and then it is determined whether the output condition is met.

[0091] The supply unit 213 supplies the one piece of message information M selected by the selection unit 212 to the output device. In this embodiment, the output device is the projection device 321 and speaker 322 of the AR glasses 30. Therefore, the supply unit 213 transmits the message information M to the wearable device 10 including the AR glasses 30. More specifically, the supply unit 213 transmits the message information M to the terminal device 40 via the communication device 203. The terminal device 40 transmits the message information M transmitted from the supply unit 213 to the AR glasses 30 via the communication device 403. The processing device 328 of the AR glasses 30 controls the projection device 321 and speaker 322 based on the received message information M, and outputs the message information M. An example of an output mode of the message information M is shown in FIG. 11 .

[0092] In this embodiment, one piece of message information M refers to the message information M of one record D among the multiple records D1 to D6 contained in the message information database DB2. For example, one piece of message information M may include multiple sentences, or may include different types of information such as text information and video.

[0093] A-8. Operation of the processing unit 206 8 is a flowchart showing the operation of the processing device 206. The processing device 206 functions as the acquisition unit 210 and acquires the measured shape information, captured image information, and position information transmitted from the AR glasses 30 (step S101). The processing device 206 functions as the identification unit 211 and identifies the product name B of the product P present in the user's field of view (step S102). Specifically, the processing device 206 performs at least one of matching the reference shape information 502 stored in the object identification database DB1 with the measured shape information or matching the reference image information 504 stored in the object identification database DB1 with the captured image information, thereby identifying the product name B of the product P present in the user's field of view. As described above, the product name B identified by the identification unit 211 is referred to as a specific product name BX. Note that if the product P is not in the user's field of view, the processing device 206 waits until the product P enters the user's field of view.

[0094] The processor 206 functions as the selection unit 212 and determines whether the specific product name BX is registered in the message information database DB2 (step S103). Specifically, the processor 206 determines whether a record D including the specific product name BX exists in the product name B of the message information database DB2. If the specific product name BX is not registered in the message information database DB2 (step S103: NO), the processor 206 returns the process to step S101 and repeats the subsequent processes.

[0095] If the specific product name BX is registered in the message information database DB2 (step S103: YES), the processing device 206 functions as the selection unit 212 and determines whether or not the record D including the specific product name BX includes a record D including the location information of the AR glasses 30 (step S104). More specifically, the processing device 206 references the location information 601 of the record D including the specific product name BX and determines whether or not there is a record D that stores location information that matches the location information of the AR glasses 30 or that includes the location information of the AR glasses 30. If there is no record D including the location information of the AR glasses 30 (step S104: NO), the processing device 206 returns the process to step S101 and repeats the subsequent processes.

[0096] If there is record D including the position information of the AR glasses 30 (step S104: YES), the processing device 206 functions as the acquisition unit 210 and acquires environmental information or time information corresponding to the output condition of record D including the position information of the AR glasses 30 (step S105). More specifically, the processing device 206 refers to the output condition information 602 of record D including the position information of the AR glasses 30, determines the type of information required to determine the output condition (for example, temperature information, time information, etc.), and acquires the required information.

[0097] When the environmental information or time information is acquired, the processing device 206 functions as the selection unit 212 and determines whether there is a record D in which the state indicated by the environmental information or time information matches the output condition (step S106). If there is no record D that matches the output condition (step S106: NO), the processing device 206 returns the process to step S101 and repeats the subsequent processes.

[0098] If there is a record that matches the output condition (step S106: YES), the processing device 206 functions as the supply unit 213 and transmits message information M corresponding to the record that matches the output condition to the AR glasses 30 (step S107). Upon receiving the message information M, the AR glasses 30 output the message information M using the projection device 321 and the speaker 322. Thereafter, the processing device 206 returns the process to step S101 and repeats the subsequent processes.

[0099] A-9. Summary of the first embodiment As described above, according to the first embodiment, the information providing server 20 selects one piece of message information M from multiple pieces of message information M based on the product name B of the product P present in the user's field of vision and related information related to the product P. The related information includes at least one of environmental information about the environment at the location where the product P is located and time information indicating the time when the product name B was identified, as well as location information indicating the location where the product P is located. Therefore, the content of the message information M is changed in accordance with the related information including the location information of the product P. Therefore, more useful information can be provided to the user compared to when a single piece of message information M is associated with the product name B.

[0100] Furthermore, according to the first embodiment, the information providing server 20 identifies the product name B by comparing measured shape information indicating the results of measuring the three-dimensional shape of the product P with reference shape information 502 indicating the three-dimensional shape of a reference object. Therefore, compared to when only captured image information is used, erroneous recognition, such as recognizing a photo of a poster as an actual object, is prevented, and display of unnecessary information is avoided.

[0101] Furthermore, according to the first embodiment, the information providing server 20 identifies the product name B by comparing the measured shape information with the reference shape information 502, and also by comparing captured image information of the product P within the user's field of view with reference image information 504 of the external appearance of the reference object. Therefore, even if the product P is an object that cannot be identified by shape alone, such as a bagged snack, it can be correctly detected.

[0102] Furthermore, according to the first embodiment, the information providing server 20 acquires measured shape information from the 3D scanner 325 of the AR glasses 30 worn on the user's head, and transmits message information M to the AR glasses 30. Therefore, the product P in the user's field of view is accurately detected in real time, and message information M appropriate for the conditions around the user is provided.

[0103] B. Second embodiment A second embodiment of the present invention will be described below. In the following description, for the sake of simplicity, the same components as those in the first embodiment will be denoted by the same reference numerals, and a description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0104] In the first embodiment, the related information was at least one of environmental information about the environment at the location where the product P was located and time information indicating the time when the identification unit 211 identified the product name B, and location information indicating the location where the product P was located. In the second embodiment, a case will be described in which the related information includes, in addition to the above information, area information about an area on the surface of the product P that is visually recognized by the user. That is, in the second embodiment, the related information includes at least one of area information, environmental information, and time information about an area on the surface of the product P that is visually recognized by the user, and location information.

[0105] FIG. 10 is an explanatory diagram that schematically illustrates the surface area of ​​a product P. In FIG. 10, a PET bottled beverage 70 with product name B "AA Soda" is illustrated as the product P. The PET bottled beverage 70 has a lid 72 and a body 74, and a label 78 is affixed to the body 74. It is rare for the label 78 of the PET bottled beverage 70 to have the same markings all around. Therefore, the surface area of ​​the PET bottled beverage 70 that is visually recognized by the user can be identified based on the markings on the label 78, for example.

[0106] Generally, the label 78 of a PET bottled beverage 70 has a first side 710 on which a product name 702 is displayed in large letters, and a second side 720 on which product information 704, such as ingredients, manufacturer, and nutritional information, and a barcode 706 are displayed. When the label 78 of the PET bottled beverage 70 shown in an image captured by the imaging device 324 of the AR glasses 30 worn by a user, faces the first side 710, it can be said that the user is viewing a first region F1 of the surface of the PET bottled beverage 70. When the label 78 of the PET bottled beverage 70 shown in the captured image faces the second side 720, it can be said that the user is viewing a second region F2 of the surface of the PET bottled beverage 70. Note that which region of the surface of the product P the user is viewing can be determined, for example, based on captured image information.

[0107] The area of ​​the surface of the PET bottled beverage 70 that the user sees is not limited to the first area F1 or the second area F2. The user can see the PET bottled beverage 70 from various angles, including the boundary between the first surface 710 and the second surface 720, and the area may be defined more precisely. For ease of explanation, this embodiment will be described assuming that the area of ​​the surface of the PET bottled beverage 70 that the user sees is either the first area F1 or the second area F2.

[0108] Fig. 9 is an explanatory diagram showing an example of the message information database DB2. As in the first embodiment, the message information database DB2 includes a product name B, location information 601, output condition information 602, and message information M. The message information database DB2 shown in Fig. 9 includes four records D1 to D4.

[0109] 9, the product name B of records D1 to D4 is “AA Soda.” Furthermore, the location information 601 of records D1 and D2 is “CC Mart DD Branch,” and the location information 601 of records D3 and D4 is “CC Mart EE Branch.”

[0110] In the second embodiment, the output condition information 602 is area information relating to an area of ​​the surface of the product P that is visually recognized by the user. Specifically, the output condition information 602 of records D1 and D3 is "first area F1," and the output condition information 602 of records D2 and D4 is "second area F2." As in the first embodiment, a plurality of types of states may be combined as the output condition information. For example, an output condition of "first area F1" and "time 10:00 to 15:59" may be set.

[0111] For example, assume that the user is at "CC Mart DD Branch" and sees "AA Soda," a PET bottled beverage 70. This state corresponds to record D1 or D2 in the message information database DB2 shown in FIG.

[0112] When the user looks at the first side 710 of the label 78 of the PET bottled beverage 70, the area of ​​the surface of the PET bottled beverage 70 that the user sees becomes the first area F1. That is, the output condition of record D1 is met. In this case, message information M corresponding to record D1 is transmitted to the AR glasses 30, and the AR glasses 30 display the transmitted message information M. Specifically, as shown in FIG. 11, text C1 saying "10% off now" is displayed.

[0113] Furthermore, when the user looks at the second side 720 of the label 78 of the PET bottled beverage 70, the area of ​​the surface of the PET bottled beverage 70 that the user sees becomes the second area F2. That is, the output condition of record D2 is met. In this case, message information M corresponding to record D2 is transmitted to the AR glasses 30, and the AR glasses 30 display the transmitted message information M. Specifically, as shown in FIG. 12, text C2 indicating the ingredients and nutritional information of the AA soda is displayed.

[0114] For example, when a user is looking at the first side 710 of the label 78, it is considered that the user is interested in the product itself. By outputting information that motivates the user to purchase, such as "10% off," at such a timing, it is possible to increase the likelihood that the user will purchase product P. Furthermore, when a user is looking at the second side 720 of the label 78, it is considered that the user is interested in the ingredients or nutritional components of product P. By outputting the ingredients and nutritional components at such a timing, the user can obtain information that will be useful in selecting a product. This improves user convenience.

[0115] As described above, according to the second embodiment, the message information M is selected based on area information relating to the area of ​​the surface of the product P that is visually recognized by the user. Therefore, information is provided taking into consideration the user's interest in the product P, and the usefulness of the message information M output by the AR glasses 30 is improved.

[0116] In the first and second embodiments, the case where the object present in the user's field of view is a product P has been described, but the present invention is not limited to this and can be applied to objects of various sizes. Also, in the first and second embodiments, the message information M is information displayed to promote the sales of the product P sold in the store, but the message information M is not limited to this and various information can be set.

[0117] For example, the object may be a structure of an attraction in a theme park. In this case, when a user views a structure in the theme park, different message information M is displayed depending on the user's location, the time of day, or the weather conditions. The message information M may be text information such as the waiting time for the attraction or a description of the attraction, or a video showing a show held inside or around the attraction. Alternatively, for example, an image of the appearance of the structure at a different time of day (for example, the appearance of the structure lit up at night) may be displayed as message information M, superimposed on the structure in real space.

[0118] The object may also be, for example, a model of a dinosaur or animal exhibited in a museum. In this case, when a user looks at a model in the museum, different message information M is displayed depending on the user's location, the time of day, or the weather conditions. The message information M may be, for example, text information such as an explanation of the dinosaur or animal shown by the model, or audio information such as the sound of the dinosaur or animal shown by the model.

[0119] That is, the present invention is not limited to sales promotion of the product P, but can be applied to various uses, such as tourist guides at tourist spots, facility guides at facilities, explanations of exhibits, and the like.

[0120] C: Modified Example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0121] C1: First modified example In the first and second embodiments, the AR glasses 30 and the terminal device 40 are separate entities. However, this is not limiting, and for example, the AR glasses 30 may have the functions of the terminal device 40. That is, the communication device 323 of the AR glasses 30 may be connectable to the communication network N, and the position information, measured shape information, and captured image information may be transmitted directly from the AR glasses 30 to the information providing server 20. Note that, if the AR glasses 30 have the functions of the terminal device 40, another user input / output device, such as a controller or hand tracking, or another user input / output device function may be provided instead of the touch panel 401.

[0122] According to the first modification, for example, it is possible to provide the message information M to the user without using the terminal device 40.

[0123] C2: Second modified example In the first and second embodiments, the information providing server 20 identifies the product name B and selects the message information M. That is, the information providing server 20 has the identification unit 211, the selection unit 212, the object identification database DB1, and the message information database DB2. However, without being limited to this, for example, a first server having a configuration corresponding to the identification unit 211 and the object identification database DB1 and identifying the product name B, and a second server having a configuration corresponding to the selection unit 212 and the message information database DB2 and selecting the message information M may be provided.

[0124] In this case, for example, the terminal device 40 transmits measured shape information and captured image information to the first server. The first server compares the measured shape information and captured image information with the object identification database DB1 to identify the product name B. When the first server detects the product name B, it notifies the terminal device 40 of the product name B. Upon receiving the product name B notification, the terminal device 40 transmits the product name B and the location information of the AR glasses 30 to the second server. The second server compares the product name B and the location information of the AR glasses 30 with the message information database DB2. If there is a record D in which the product name B matches the location information of the AR glasses 30, the second server acquires environmental information or time information of the location of the AR glasses 30 based on the output condition information of the record D. If the output condition information of the record D matches the environmental information or time information of the location of the AR glasses 30, the second server transmits message information M corresponding to the record D to the terminal device 40.

[0125] According to the second modification, it is possible to prevent the processing load from concentrating on one server, thereby improving the stability of the system.

[0126] C3: Third modified example In the first and second embodiments, the information providing server 20 identifies the product name B. That is, the information providing server 20 has the identification unit 211 and the object identification database DB1. However, the present invention is not limited to this, and for example, the terminal device 40 may have the identification unit 211 and the object identification database DB1.

[0127] According to the third modification, product name B is identified by terminal device 40, so there is no need to transmit and receive measurement shape information and captured image information via communication network N, and the processing load on the system can be reduced.

[0128] D:Other (1) Each function illustrated in Figures 3, 4, and 5 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or multiple devices.

[0129] (2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

[0130] (3) In each of the first embodiment, the second embodiment, and the first to third modifications, storage device 205, storage device 327, and storage device 405 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Also, the program may be transmitted from a network via a telecommunications line.

[0131] (4) Each of the first embodiment, the second embodiment, and the first to third modifications may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wideband), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of LTE and / or LTE-A with 5G).

[0132] (5) The order of the exemplary processing procedures, sequences, or flowcharts shown in the first embodiment, second embodiment, and first to third modifications may be changed as long as there is no contradiction. For example, the methods described herein present various step elements in an exemplary order and are not limited to the particular order presented.

[0133] (6) In each of the first embodiment, the second embodiment, and the first to third modifications, input and output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added. Output information, etc. may be deleted. Input information, etc. may be transmitted to another device.

[0134] (7) In each of the first embodiment, the second embodiment, and the first to third variants, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0135] (8) The programs exemplified in the first embodiment, the second embodiment, and the first to third modifications should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, hardware description languages, or by other names. Furthermore, software, instructions, or the like, may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared and microwave), at least one of these wired and wireless technologies is included within the definition of a transmission medium.

[0136] (9) The information and the like described in each of the first embodiment, the second embodiment, and the first to third modifications may be represented using any of a variety of different technologies. For example, data, information, and the like that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0137] (10) In each of the first embodiment, the second embodiment, and the first to third modifications, the terms "system" and "network" are used interchangeably.

[0138] (11) In each of the first embodiment, the second embodiment, and the first to third modifications, the terminal device 40 may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term.

[0139] (12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a mobile object, or the mobile object itself. A mobile object refers to an object that can move. A mobile object can move at any speed. A mobile object can be stopped. Examples of mobile objects include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile object may also be an object that moves autonomously based on an operation command. A mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

[0140] (13) In each of the first embodiment, the second embodiment, and the first to third modifications, the term "determining" or "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), ascertaining, and regarding that as a "determination." Also, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as a "judging" or "determining." Furthermore, "decision" can include the act of considering something to be "decided" such as resolving, selecting, choosing, establishing, or comparing. In other words, "decision" can include the act of considering something to be "decided" to be an action. "Decision" can also be interpreted as "assuming," "expecting," or "considering," among others.

[0141] (14) In each of the first embodiment, the second embodiment, and the first to third modifications, the term "connected," or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0142] (15) In each of the first embodiment, the second embodiment, and the first to third modifications, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0143] (16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0144] (17) When the words "include," "including," and variations thereof are used in the first embodiment, second embodiment, and first to third modifications in this specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this specification or claims is not intended to mean an exclusive logical OR.

[0145] (18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

[0146] (19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative explanation and does not have any limiting meaning on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined. [Explanation of symbols]

[0147] 1...information processing system, 10...wearable device, 20...information providing server, 30...AR glasses, 40...terminal device, 201...input device, 202...display device, 203...communication device, 205...storage device, 206...processing device, 210...acquisition unit, 211...identification unit, 212...selection unit, 213...supply unit, 321...projection device, 322...speaker, 323...communication device, 324...imaging device, 325...3D scanner, 326...GPS device, 327...storage device, 328...processing device, DB1...database for object identification, DB2...message information database, N...communication network.

Claims

1. an identification unit that identifies a type of an object present in the user's field of view and outputs type information indicating the identified type; a selection unit that selects one piece of message information from a plurality of pieces of message information based on the type information and related information related to the object; a supply unit that supplies the selected piece of message information to an output device, the related information includes at least one of environmental information about an environment at a location where the object is present and time information indicating a time when the identification unit identified the type of the object, and location information indicating a location where the object is present; The identification unit comparing measured shape information indicating the results of measuring the three-dimensional shape of the object with reference shape information indicating the three-dimensional shape of a reference object defined for each type; comparing a first image obtained by imaging the object with a second image obtained by imaging the appearance of the reference object; If there is a matching object for which the degree of match between the measured shape information and the reference shape information is equal to or greater than a first predetermined value and the degree of match between the first image and the second image is equal to or greater than a second predetermined value, it is determined that output of the one message information is necessary, and the type of the object is identified by identifying the name of the matching object; For a first object for which the degree of coincidence between the first image and the second image is equal to or greater than the second predetermined value, if reference shape information of the first object indicates a three-dimensional shape and measured shape information of the first object indicates a plane, it is determined that output of the one message information is unnecessary. Information processing device.

2. The information processing device according to claim 1 , wherein the related information includes at least one of area information relating to an area of ​​the surface of the object that is visually recognized by the user, the environmental information, and the time information, and the position information.

3. the specifying unit acquires the measurement shape information output from a measurement device and the first image output from an imaging device; the supply unit transmits the one message information to the output device; the measuring device, the imaging device, and the output device are provided in a wearable device that is worn on the head of the user; The information processing device according to claim 1 .

4. the message information includes at least one of image information and audio information, the output device includes a see-through display disposed between the user's eyes and the object and a speaker that outputs sound to the user; the image information is output from the transmissive display; The audio information is output from the speaker. The information processing device according to claim 1 .

Images