Information processing device, information processing system, information processing method, and recording medium

Abstract

An information processing device according to the present invention comprises: an acquisition means for acquiring biological information of a subject; and a display control means for displaying a first image including products on a head-mounted display which is worn by the subject, wherein the display control means displays the first image in which the products are laid out on the basis of registration information that pertains to the subject and that has been identified using the biological information.

Description

Information processing device, information processing system, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing system, an information processing method, and a recording medium.

[0002] Services are provided to users using a display device worn on the user's head. For example, Patent Literature 1 discloses a product search device that acquires an image of a product and information on the user's line of sight with respect to a shelf on which the product is placed, recognizes the product based on the image of the product and the line of sight information, and acquires and outputs information on the recognized product.

[0003] International Publication No. 2023 / 187866

[0004] An object of this disclosure is to provide an information processing device, an information processing system, an information processing method, and a recording medium that aim to improve the technology related to the above-mentioned prior art documents.

[0005] One aspect of an information processing device includes an acquisition means for acquiring biometric information of a subject, and a display control means for displaying a first image including a product on a head-mounted display worn by the subject, wherein the display control means displays the first image in which the product is laid out based on the registration information of the subject identified using the biometric information.

[0006] One aspect of the information processing system includes a head-mounted display worn by a subject, an acquisition means for acquiring biometric information of the subject, and a display control means for displaying a first image including a product on the head-mounted display, wherein the display control means displays the first image in which the product is laid out based on registration information of the subject identified using the biometric information, and a second information processing device including an authentication means for performing biometric authentication of the subject using the biometric information and an image generation means for generating the first image based on the registration information of the subject identified by the biometric authentication.

[0007] One aspect of the information processing method is an information processing method executed by a computer that acquires biometric information of a subject and displays a first image including a product on a head-mounted display worn by the subject, and displays the first image in which the product is laid out based on the registration information of the subject identified using the biometric information.

[0008] One aspect of the recording medium is an information processing method in which a computer acquires biometric information of a subject and displays a first image including a product on a head-mounted display worn by the subject, and a computer program is recorded on the recording medium to cause the computer to execute the information processing method, which displays the first image in which the product is laid out based on the registration information of the subject identified using the biometric information.

[0009] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing a flow of a classification operation in an information processing device according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing system according to the present disclosure. FIG. 4 is a block diagram showing a configuration of a head-mounted display device according to the present disclosure. FIG. 5 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 6 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 7 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 8 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 9 is a flowchart showing a flow of a classification operation in an information processing device according to the present disclosure. FIG. 10 is a block diagram showing a configuration of an information processing device according to the present disclosure.

[0010] [1: First embodiment] A first embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to Figs. 1 and 2. Hereinafter, the first embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing device 10. [1-1: Configuration of information processing device 10]

[0011] The configuration of an information processing device 10 according to this disclosure will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing device 10 according to this disclosure. As shown in FIG. 1, the information processing device 10 includes a biometric information acquisition unit 11 and a display control unit 12. [1-2: Information Processing Method Executed by the Information Processing Device 10]

[0012] An information processing method executed by the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 10. The information processing device 10 may be realized by a computer executing a computer program for causing a computer to execute the information processing method.

[0013] 2, the biometric information acquisition unit 11 acquires biometric information of a target user (step S11). The user wears a device with a display function (referred to as a "head-mounted display") on his or her head.

[0014] The display control unit 12 displays a first image including the products on the head-mounted display worn by the user (step S12). The first image has the products laid out based on the registered information of the user identified using the biometric information. The registered information is registered information about the user. The registered information includes information registered by the user himself / herself.

[0015] The registration information may include, for example, the user's biometric information, information indicating the user's attributes, and information regarding the user's payment. The user's attributes may include, for example, the user's age, gender, place of residence, family structure, occupation, etc. The user's payment information may include, for example, the user's bank account information, credit card information, electronic money information, etc.

[0016] The layout of the products may include adjusting the display location of the products. For example, the first image may be laid out so that the products that are estimated to be more likely to be purchased by the user based on the registration information are displayed in a location that is more likely to catch the user's eye. The location that is more likely to catch the user's eye may be the center of the first image.

[0017] The layout of the products may include adjusting the display size of the products. For example, the first image may be laid out so that the size of the products that are estimated to be likely to be purchased by the user based on the registration information is displayed larger than the size of the products that are estimated to be unlikely to be purchased by the user. [1-3: Technical Effects]

[0018] The information processing device 10 according to this disclosure can provide a product layout suitable for the user. [2: Second Embodiment]

[0019] A second embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to FIGS. 3 to 9. In the following, the second embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing system 1000. Note that, for the second embodiment, descriptions that overlap with the first embodiment will be omitted as appropriate. [2-1: Configuration of Information Processing System 1000]

[0020] The configuration of an information processing system 1000 as an information processing apparatus in this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing system 1000 in this embodiment.

[0021] As shown in FIG. 3 , the information processing system 1000 includes a head-mounted display device 100 and a store device 200. The head-mounted display device 100 is a form of a first information processing device. The store device 200 is a form of a second information processing device. The head-mounted display device 100 is a device with a display function that is worn on the user's head. The information processing system 1000 is configured to be able to provide the user with a world created in a virtual space on the Internet via the head-mounted display device 100. The information processing system 1000 is configured to be able to provide the user with a shopping experience, particularly at a store (referred to as a "virtual store") created in a virtual space on the Internet, via the head-mounted display device 100. In other words, the user can enter the virtual store via the head-mounted display device 100.

[0022] The in-store device 200 may be, for example, a server device. The server device may be a cloud server. The in-store device 200 may also be, for example, a personal computer. In other words, the head-mounted display device 100 and the in-store device 200 constitute a single information processing system 1000. The head-mounted display device 100 and the in-store device 200 are configured to be able to communicate with each other. [2-2: Configuration of the head-mounted display device 100]

[0023] The configuration of the head-mounted display device 100 in this embodiment will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a schematic diagram showing the configuration of the head-mounted display device 100 in this embodiment. Fig. 5 and Fig. 6 are block diagrams showing the configuration of the head-mounted display device 100 in this embodiment.

[0024] 4, in this embodiment, an immersive head-mounted display (so-called immersive VR (Virtual Reality) goggles) is given as an example of the head-mounted display device 100. Note that the head-mounted display device 100 is not limited to an immersive head-mounted display, and may be, for example, a video see-through head-mounted display or an optical see-through head-mounted display (so-called AR (Augmented Reality) glasses).

[0025] 5 , the head mounted display device 100 includes a calculation device 110, a storage device 120, a communication device 130, an input device 140, an output device 150, and cameras 171 and 172. The calculation device 110, the storage device 120, the communication device 130, the input device 140, the output device 150, and the cameras 171 and 172 may be connected via a data bus 160.

[0026] The computing device 110 may include a processor 110a. The computing device 110 may include other processors in addition to the processor 110a. That is, the computing device 110 may include one or more processors. The processor 110a may be a multi-core processor. When the computing device 110 includes a single processor 110a that is a multi-core processor, the computing device 110 can be said to logically include multiple processors.

[0027] The processor 110a may be, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processing unit (TPU), and a quantum processor.

[0028] The storage device 120 may include a memory 120a. The storage device 120 may include other memories in addition to the memory 120a. That is, the storage device 120 may include one or more memories. The memory 120a may be, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. Therefore, the storage device 120 may include the memory 120a as a non-transitory recording medium.

[0029] The storage device 120 can store desired data. A computer program 121 to be executed by the arithmetic device 110 may be stored in the memory 120a of the storage device 120. The storage device 120 may temporarily store data that is temporarily used by the arithmetic device 110 when the arithmetic device 110 is executing the computer program 121.

[0030] The computer program 121 may be recorded on a computer-readable, non-transitory recording medium. In this case, the computer program 121 may be stored in the memory 120a by reading the recording medium using a recording medium reading device (not shown) included in the head-mounted display device 100. The recording medium may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program. The computer program 121 may be acquired (in other words, downloaded) from a device (not shown) external to the head-mounted display device 100 via the communication device 130. The acquired computer program 121 may be stored in the memory 120a.

[0031] The communication device 130 may be capable of communicating with a device external to the head mounted display device 100. The communication device 130 may perform wired communication or wireless communication. Typically, the head mounted display device 100 may be configured to be capable of communicating with the store device 200 via the communication device 130.

[0032] The input device 140 is a device capable of receiving information input to the head-mounted display device 100 from an external device. The input device 140 may include a recording medium reading device capable of reading information recorded on a recording medium detachable from the head-mounted display device 100, such as a USB (Universal Serial Bus) memory. When information is input to the head-mounted display device 100 via the communication device 130 (in other words, when the head-mounted display device 100 acquires information via the communication device 130), the communication device 130 may function as an input device. When information is input to the head-mounted display device 100 via the cameras 171 and 172 (in other words, when the head-mounted display device 100 acquires information via the cameras 171 and 172), the cameras 171 and 172 may function as input devices. In this case, the head-mounted display device 100 may acquire information indicating the user's eye movements based on images captured by the cameras 171 and 172. The user's eye movement may be, for example, at least one of the user's line of sight and the user's wink. The input device 140 may include an operation device (e.g., a remote control, a controller, a keyboard, a mouse, a touch panel, etc.) that can be operated by the user of the head-mounted display device 100.

[0033] The output device 150 is a device capable of outputting information to the outside of the head-mounted display device 100. The output device 150 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 150 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The display may be, for example, at least one of a liquid crystal display and an organic electroluminescence (EL) display. The printer may be, for example, at least one of an inkjet printer and a laser printer. The output device 150 may be capable of outputting information to a recording medium detachable from the head-mounted display device 100, such as a USB memory. When the head-mounted display device 100 outputs information via the communication device 130, the communication device 130 may function as the output device.

[0034] 4, the head-mounted display device 100 includes a display 1521 as an output device 150, a lens 1522L for the left eye, a lens 1522R for the right eye, and a plurality of light sources 1523#1, 1523#2, 1523#3, and 1523#4 that emit near-infrared light. The light sources 1523#1 and 1523#2 may be arranged around the lens 1522L. The light sources 1523#3 and 1523#4 may be arranged around the lens 1522R. The lenses 1522L and 1522R may be referred to as VR lenses.

[0035] The head mounted display device 100 may have a display for the left eye and a display for the right eye instead of the display 1521. The number of light sources that emit near-infrared light is not limited to four, and may be five or more, or three or less.

[0036] The camera 171 may be arranged in the head mounted display device 100 so as to be able to capture an image of the user's left eye. The camera 172 may be arranged in the head mounted display device 100 so as to be able to capture an image of the user's right eye. The cameras 171 and 172 may be cameras capable of capturing near-infrared light. In other words, the cameras 171 and 172 may be cameras equipped with imaging elements sensitive to the near-infrared wavelength range. The cameras 171 and 172 may constitute part of the head mounted display device 100. In other words, the head mounted display device 100 may have the cameras 171 and 172.

[0037] The processor 110a of the arithmetic device 110, together with the memory 120a of the storage device 120 in which a computer program 121 is stored (in other words, together with the memory 120a and the computer program 121 stored in the memory 120a), may execute processing to be performed by the head mounted display device 100. For example, by the processor 110a executing the computer program 121, a logical functional block for executing processing to be performed by the head mounted display device 100 may be realized in the arithmetic device 110 (for example, in the processor 110a).

[0038] As shown in FIG. 6 , the arithmetic device 110 may have an image acquisition unit 111 and a display control unit 112 as logically realized functional blocks or as physically realized processing circuits. At least one of the image acquisition unit 111 and the display control unit 112 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. The "image acquisition unit 111" and the "display control unit 112" are components corresponding to the "biometric information acquisition unit 11" and the "display control unit 12" in the first embodiment, respectively. The processes performed by the image acquisition unit 111 and the display control unit 112 will be described with reference to FIG. 9 .

[0039] When the image acquisition unit 111 and the display control unit 112 are realized as functional blocks, the image acquisition unit 111 and the display control unit 112 may be realized by a single processor (for example, the processor 110a), or the image acquisition unit 111 and the display control unit 112 may be realized by different processors.

[0040] The head-mounted display device 100 may be a standalone type head-mounted display. Alternatively, the head-mounted display device 100 may be a PC-connected type head-mounted display connected to a personal computer. In this case, the arithmetic unit 110, the storage device 120, the communication device 130, the input device 140, the output device 150, and the cameras 171 and 172 may be included in the personal computer to which the head-mounted display device 100 is connected. The personal computer and the store device 200 may constitute a single information processing device. The head-mounted display device 100 may be a hybrid type head-mounted display (i.e., a head-mounted display that can operate as both a standalone type and a PC-connected type). [2-3: Configuration of the store device 200]

[0041] The configuration of the in-store device 200 in this embodiment will be described with reference to Figures 7 and 8. Figures 7 and 8 are block diagrams showing the configuration of the in-store device 200 in this embodiment.

[0042] 7 , the in-store device 200 includes a computing device 210, a storage device 220, a communication device 230, an input device 240, and an output device 250. The computing device 210, the storage device 220, the communication device 230, the input device 240, and the output device 250 may be connected via a data bus 260.

[0043] The arithmetic device 210 may include a processor 210a. The arithmetic device 210 may include other processors in addition to the processor 210a. That is, the arithmetic device 210 may include one or more processors. The processor 210a may be a multi-core processor. When the arithmetic device 210 includes a single processor 110a that is a multi-core processor, it can be said that the arithmetic device 210 logically includes multiple processors.

[0044] The processor 210a may be, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processing unit (TPU), and a quantum processor.

[0045] The storage device 220 may include a memory 220a. The storage device 220 may include other memories in addition to the memory 220a. That is, the storage device 220 may include one or more memories. The memory 220a may be, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. Therefore, the storage device 220 may include the memory 220a as a non-transitory recording medium.

[0046] The storage device 220 can store desired data. A computer program 221 to be executed by the arithmetic device 210 may be stored in the memory 220a of the storage device 220. The storage device 220 may temporarily store data that is temporarily used by the arithmetic device 210 when the arithmetic device 210 is executing the computer program 221.

[0047] The computer program 221 may be recorded on a computer-readable, non-transitory recording medium. In this case, the computer program 221 may be stored in the memory 220a by reading the recording medium using a recording medium reading device (not shown) included in the in-store device 200. The recording medium may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program. The computer program 221 may be acquired (in other words, downloaded) from a device (not shown) external to the in-store device 200 via the communication device 230. The acquired computer program 221 may be stored in the memory 220a.

[0048] The communication device 230 may be capable of communicating with a device external to the in-store device 200. The communication device 230 may perform wired communication or wireless communication. Typically, the in-store device 200 may be configured to be capable of communicating with the head-mounted display device 100 via the communication device 230.

[0049] The input device 240 is a device capable of accepting information input to the in-store device 200 from an external device. The input device 240 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the in-store device 200. The input device 240 may include a recording medium reading device capable of reading information recorded on a recording medium that is detachable from the in-store device 200, such as a USB (Universal Serial Bus) memory. Note that when information is input to the in-store device 200 via the communication device 130 (in other words, when the in-store device 200 obtains information via the communication device 230), the communication device 230 may function as an input device.

[0050] The output device 250 is a device capable of outputting information to the outside of the in-store device 200. The output device 250 may output visual information such as text or images, or may output auditory information such as sound, as the information. The output device 250 may include, for example, at least one of a display, a speaker, and a printer. The output device 250 may be capable of outputting information to a recording medium that is detachable from the in-store device 200, such as a USB memory. Note that when the head-mounted display device 100 outputs information via the communication device 230, the communication device 230 may function as the output device.

[0051] The processor 210a of the arithmetic device 210, together with the memory 220a of the storage device 220 in which the computer program 221 is stored (in other words, together with the memory 220a and the computer program 221 stored in the memory 220a), may execute the processing to be performed by the in-store device 200. For example, the processor 210a may execute the computer program 221 to realize logical function blocks in the arithmetic device 210 (e.g., in the processor 210a) for executing the processing to be performed by the in-store device 200.

[0052] 8, the computing device 210 may have a biometric authentication unit 213, an image generation unit 214, and a registration information acquisition unit 215 as logically realized functional blocks or as physically realized processing circuits. At least one of the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. The processing performed by each of the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 will be described with reference to FIG. 9.

[0053] When the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 are realized as functional blocks, the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be realized by a single processor (e.g., the processor 210a). Alternatively, the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be realized by different processors. Alternatively, parts of the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be realized by a single processor, and the remaining parts of the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be realized by one or more processors different from the single processor.

[0054] Furthermore, the computing device 210 may not have the biometric authentication unit 213. In this case, the in-store device 200 may be configured to be able to communicate with an authentication device different from the in-store device 200 via the communication device 230. In other words, biometric authentication of a user may be performed by an authentication device different from the in-store device 200. [2-4: Information Processing Method Executed by the Information Processing System 1000]

[0055] An information processing method executed by the information processing system 1000 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000.

[0056] 9 , the image acquisition unit 111 acquires biometric information of a user (step S21). The user wears the head-mounted display device 100 on his or her head. The biometric information of the user may be, for example, an eye image including the user's eyes, which is generated by capturing an image of the user's eyes with at least one of the cameras 171 and 172.

[0057] The biometric authentication unit 213 performs biometric authentication of the user using the biometric information (step S22). The biometric authentication unit 213 may perform at least one of iris authentication, face authentication, fingerprint authentication, palm print authentication, voice print authentication, and ear acoustic authentication as the biometric authentication of the user. Furthermore, the biometric authentication unit 213 may perform at least one of face authentication, fingerprint authentication, palm print authentication, voice print authentication, and ear acoustic authentication in addition to iris authentication, for example. In other words, the biometric authentication unit 213 may perform multimodal biometric authentication as the biometric authentication of the user. The biometric authentication unit 213 identifies the user through biometric authentication.

[0058] The registration information acquisition unit 215 acquires the registration information of the identified user (step S23). For example, the registration information acquisition unit 215 may acquire information indicating the attributes (e.g., gender, age, etc.) of users who visit the virtual store. The image generation unit 214 generates a first image based on the user's registration information (step S24). For example, if the virtual store is a food store and the user is a young man, the image generation unit 214 may generate a first image in which products are laid out so that high-calorie products are the main focus. Alternatively, if the user is a young woman, the image generation unit 214 may generate a first image in which products are laid out so that low-calorie products are the main focus.

[0059] The display control unit 112 displays the first image on the display 1521 (step S25). The first image may dynamically change in response to the user's actions. The first image may be a three-dimensional image. The first image may be a moving image.

[0060] In the above-described embodiment, among the functions of the image acquisition unit 111, the display control unit 112, the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 included in the information processing system 1000 as an information processing device, the image acquisition unit 111 and the display control unit 112 are implemented in the head-mounted display device 100, and the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 are implemented in the store device 200. However, this is merely an example. That is, the functions of the image acquisition unit 111, the display control unit 112, the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be implemented in the head-mounted display device 100 or in the store device 200. For example, the functions of the information processing system 1000 may be implemented as in the following Modifications 1 to 3. The same applies to other embodiments in which the functions of the information processing system may be implemented in either the head-mounted display device or the store device. [2-5-1: Modification 1]

[0061] The functions of the biometric authentication unit 213, the image generation unit 214, and the registration information acquisition unit 215 may be implemented within the arithmetic unit 110 of the head-mounted display device 100. In this case, the head-mounted display device 100 identifies the user without communicating with the store device 200, and displays the first image in a layout suitable for the user. [2-5-2: Modification 2]

[0062] The functions of the image generation unit 214 and the registration information acquisition unit 215 may be realized within the arithmetic device 110 of the head mounted display device 100. In this case, the arithmetic device 110 of the head mounted display device 100 transmits the biometric information of the user acquired by the image acquisition unit 111 to the store device 200 via the communication device 130. The store device 200 transmits the result of the biometric authentication to the head mounted display device 100. [2-5-3: Modification 3]

[0063] The functions of the image acquisition unit 111 and the display control unit 112 may be implemented within the computing device 210 of the in-store device 200. In this case, the head-mounted display device 100 displays the first image on the display 1521 under the control of the in-store device 200. [2-6: Technical Effects]

[0064] The information processing system 1000 according to this disclosure can provide a product layout in a virtual store that is suitable for the user. [3: Third Embodiment]

[0065] A third embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to FIGS. 10 and 11. Hereinafter, the third embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing system 1000a. Note that, for the third embodiment, descriptions that overlap with the first and second embodiments will be omitted as appropriate. [3-1: Configuration of Information Processing System 1000a]

[0066] The configuration of an information processing system 1000a as an information processing device in this embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the information processing system 1000a in this embodiment. As shown in Fig. 10, the information processing system 1000a includes a head-mounted display device 100 and a store device 300. [3-2: Registration Information]

[0067] In this embodiment, the registration information includes information indicating at least one of preferred items and items to be avoided for the user and / or related individuals associated with the user. The registration information may be registered in a registration information database DB1 implemented in the storage device 220. Related individuals include the user's family. In this embodiment, individuals living with the user are referred to as family members. Family members include pets. Note that family members do not have to be living beings. When a user wants to purchase a product for an individual on a daily basis, the user can register the relevant individual as a related individual. Hereinafter, the above-mentioned "related individuals" will be referred to as "family members."

[0068] Products available for purchase in the store are associated with product information related to the product. The product information may be registered in a product information database DB2 implemented in the storage device 220. The product information may include at least one of price information, size information, weight information, color information, raw material information, ingredient information, country of origin information, place of origin information, manufacturer information, seller information, importer information, production date information, best-before date information, expiration date information, calorie information, and allergy information.

[0069] The user may register information corresponding to the product information as the registration information. For example, a user (including the user's family; hereinafter, in this embodiment, even when simply referred to as "user", the user's family may be included) who is particular about the country of origin may register the country of origin information as the registration information. In this case, the image generation unit 314 may generate the first screen based on the registered country of origin information. For example, a user with an allergy may register allergy information as the registration information. In this case, the image generation unit 314 may generate the first screen based on the registered allergy information. For example, the image generation unit 314 may generate the first screen so as not to include products that conflict with the allergy information included in the user's registration information. [3-3: Information Processing Method Executed by the Information Processing System 1000a]

[0070] An information processing method executed by the information processing system 1000a will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000a.

[0071] 11 , the image acquisition unit 111 acquires biometric information of a user (step S21). The image acquisition unit 111 acquires an eye image including the user's eyes, which is generated by at least one of the cameras 171 and 172 capturing an image of the user's eyes. The following describes an example in which the image acquisition unit 111 acquires an eye image. The image acquisition unit 111 acquires at least a part of the user's eye image as an eye image to be used for biometric authentication of the user (e.g., iris authentication).

[0072] The arithmetic device 110 of the head-mounted display device 100 may transmit the user's eye image (here, the eye image used for biometric authentication of the user) acquired by the image acquisition unit 111 to the store device 300 via the communication device 130. The store device 300 may use the eye image to perform iris authentication as an example of biometric authentication. The store device 300 may transmit the result of the biometric authentication to the head-mounted display device 100.

[0073] The biometric authentication unit 313 performs biometric authentication of the user using at least a portion of the eye image acquired by the image acquisition unit 111 (step S32). The following describes the operation of the biometric authentication unit 313 when performing iris authentication, an example of biometric authentication. The biometric authentication unit 313 may generate an iris image including an iris region corresponding to the user's iris from the eye image. The biometric authentication unit 313 may calculate a matching score by matching the generated iris image with a pre-registered iris image. Here, the matching score may refer to a score indicating the degree of match between the generated iris image and the pre-registered iris image. Note that the higher the degree of match between the generated iris image and the pre-registered iris image, the higher the matching score may be.

[0074] Alternatively, the biometric authentication unit 313 may extract features related to the generated iris image. The biometric authentication unit 313 may compare the extracted features with pre-registered features to calculate a matching score. Here, the matching score may refer to a score indicating the degree of match between the features extracted from the iris image and the pre-registered features. Note that the higher the degree of match between the features extracted from the iris image and the pre-registered features, the higher the matching score may be.

[0075] The biometric authentication unit 313 may compare the matching score with a predetermined threshold. If the matching score is equal to or greater than the predetermined threshold, the biometric authentication unit 313 may determine that the user is a person associated with the pre-registered iris image or feature amount. In this case, it can be said that a person (e.g., identification information) corresponding to the user has been identified through biometric authentication. If the matching score is less than the predetermined threshold, the biometric authentication unit 313 may determine that the user is not a person associated with the pre-registered iris image or feature amount. In this case, it can be said that a person corresponding to the user has not been identified through biometric authentication.

[0076] The biometric authentication unit 313 may calculate a matching score for the user's left eye and a matching score for the user's right eye. However, the biometric authentication unit 313 may calculate only one of the matching score for the left eye and the matching score for the right eye. In other words, the biometric authentication unit 313 may calculate at least one of the matching score for the left eye and the matching score for the right eye.

[0077] The biometric authentication unit 313 may calculate the matching score based on the degree of match between the iris image of the user's left eye and one pre-registered iris image and the degree of match between the iris image of the user's right eye and another pre-registered iris image. Alternatively, the biometric authentication unit 313 may calculate the matching score based on the degree of match between a feature extracted from the iris image of the user's left eye and one pre-registered feature and the degree of match between a feature extracted from the iris image of the user's right eye and another pre-registered feature.

[0078] The registration information acquisition unit 315 acquires the registration information of the identified user (step S33). The registration information acquisition unit 315 may acquire the registration information of the identified user from the registration information database DB1.

[0079] The image generation unit 314 generates a first image based on the user's registration information (step S34). The image generation unit 314 may generate the first image based on the user's registration information and product information registered in the product information database DB2. The image generation unit 314 lays out the products to be included in the first image based on the user's registration information so that products related to things the user should avoid are not displayed. In other words, the image generation unit 314 may generate a first image that does not include products related to things the user should avoid.

[0080] Alternatively, the image generation unit 314 may generate a first image that highlights products related to products that the user should avoid, based on the registered information. Highlighting may include displaying the relevant products in black, displaying the relevant products as transparent, or displaying the relevant products in a frame. The display control unit 112 displays the first image generated by the image generation unit 314 (step S25). The display control unit 112 may present the first image generated by the image generation unit 314 as recommendation information. [3-4: Technical Effects]

[0081] The information processing system 1000a according to this disclosure can reduce the possibility that a user will mistakenly purchase a product. In addition, the user does not need to look at unnecessary products, which improves shopping efficiency. [4: Fourth Embodiment]

[0082] A fourth embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to FIGS. 12 and 13. Hereinafter, the fourth embodiment of an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing system 1000b. Note that, for the fourth embodiment, descriptions that overlap with the first to third embodiments will be omitted as appropriate. The virtual store in this embodiment may be a department store including multiple specialty stores. [4-1: Configuration of Information Processing System 1000b]

[0083] The configuration of an information processing system 1000b as an information processing device in this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the information processing system 1000b in this embodiment. As shown in Fig. 12, the information processing system 1000b includes a head-mounted display device 100 and a store device 400. [4-2: Configuration of the store device 400]

[0084] 12, the registration information acquisition unit 415 implemented in the computing device 210 of the store device 400 may include a store identification unit 4151 and a target individual estimation unit 4152. In addition, the product information database DB2 in this embodiment may include a store product table in which stores are associated with products that can be purchased at the stores. [4-3: Information Processing Method Executed by the Information Processing System 1000b]

[0085] An information processing method executed by the information processing system 1000b will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000b.

[0086] 13, the image acquisition unit 111 acquires an image of the user's eyes (step S21). The biometric authentication unit 313 performs iris authentication of the user using the eye image (step S32).

[0087] The registration information acquisition unit 415 acquires the registration information of the identified user (step S43). The store identification unit 4151 identifies the virtual store the user is visiting in the virtual space (step S431). The target individual estimation unit 4152 estimates the target individual from the identified virtual store (step S432). That is, the target individual estimation unit 4152 estimates who the user is visiting the store for from the identified virtual store. The target individual estimation unit 4152 estimates whether the user is visiting the store for themselves or for a family member.

[0088] The image generation unit 414 generates a first image according to the store the user is visiting (step S44). For example, if the store the user is visiting is a specialty store that sells products for infants, it is considered that the user is shopping for an infant, who is a related individual of the user. In this case, for example, it is necessary to avoid products related to things that infants should avoid, but it is not necessary to avoid products related to things that the user himself should avoid. Also, if the store the user is visiting is a specialty store that sells products for pets, it is considered that the user is shopping for a pet, who is a related individual of the user. In this case, for example, it is necessary to avoid products related to things that pets should avoid, but it is not necessary to avoid products related to things that the user himself should avoid.

[0089] The image generation unit 414 changes the registration information used to generate the first image depending on the store (services such as products offered by the store) being visited, i.e., whether the user is visiting the store to purchase pet-related products or child-related products. In other words, if the user is visiting a store for infants and young children, information about pet allergies and the like may be ignored. Also, if the user is visiting a store for pets, information about children's allergies and the like may be ignored. The display control unit 112 presents the first image generated by the image generation unit 414 as recommended information (step S25). [4-4: Technical Effects]

[0090] The information processing system 1000b according to this disclosure can present a first image with a layout more suited to the user's requirements, and can provide recommended information more suited to the user's requirements. [5: Fifth Embodiment]

[0091] A fifth embodiment relating to an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to FIGS. 14 and 15. Hereinafter, the fifth embodiment relating to an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing system 1000c. Note that, for the fifth embodiment, descriptions that overlap with the first to fourth embodiments will be omitted as appropriate. [5-1: Configuration of Information Processing System 1000c]

[0092] The configuration of an information processing system 1000c as an information processing device in this embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the configuration of the information processing system 1000c in this embodiment. As shown in Fig. 14, the information processing system 1000c includes a head-mounted display device 100 and a store device 500. [5-2: Configuration of the store device 500]

[0093] 14, in addition to the biometric authentication unit 313, the image generation unit 514, and the registration information acquisition unit 315, the computing device 210 of the in-store device 500 also includes an eye tracking unit 516, an interest estimation unit 517, and a warning unit 518. The interest estimation unit 517 may include a motion detection unit 5171. [5-3: Information Processing Method Executed by the Information Processing System 1000c]

[0094] An information processing method executed by the information processing system 1000c will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000c.

[0095] 15A, the image acquisition unit 111 acquires an eye image of the user (step S21). The biometric authentication unit 313 performs iris authentication of the user using the eye image (step S32). The registration information acquisition unit 315 acquires registration information of the identified user (step S33). The image generation unit 514 generates a first image based on the user's registration information (step S34).

[0096] The display control unit 112 presents the first image generated by the image generation unit 514 as recommended information (step S25). The display control unit 112 may arrange the first image in a location in the virtual store that is easily noticeable to the user and present it as recommended information. The first image may be an image of a display shelf in the virtual store that is located in the direction that the user is looking and intending to approach. For example, if the display shelf in question is a shelf that displays beverages, the image generation unit 514 may generate a first image in which beverages are laid out according to the registration information. For example, if the user prefers vegetable-based beverages but is allergic to dairy ingredients, the image generation unit 514 may generate a first screen that places vegetable-based beverages in a location on the display shelf that is easily noticeable to the user, while not displaying beverages containing dairy ingredients. Alternatively, the image generation unit 514 may generate a first screen in which beverages containing dairy ingredients are placed in a less noticeable location on the display shelf.

[0097] The information presentation operation in step S50 will be described with reference to Fig. 15(b). As shown in Fig. 15(b), the gaze tracking unit 516 acquires an image of the user's eyes (step S51). The eye image acquired by the gaze tracking unit 516 may be the eye image acquired by the image acquisition unit 111.

[0098] The gaze tracking unit 516 tracks the user's gaze using the eye image (step S52). The gaze tracking unit 516 tracks the user's gaze (referred to as "user gaze") in order to understand the user's interests in the virtual store. In other words, the gaze tracking unit 516 measures the user's eye movement.

[0099] The gaze tracking unit 516 performs gaze tracking to track the user's gaze on the display 1521 using the eye image. The following method is an example of a gaze tracking method. It is assumed that the user's eyes are illuminated by a point light source emitting near-infrared light. The gaze tracking unit 516 may estimate and track the gaze based on the relative positional relationship between the position of near-infrared light reflected by the cornea (i.e., Purkinje image) included in the eye image and the position of the pupil included in the eye image. Alternatively, the gaze tracking unit 516 may estimate and track the gaze based on the relative positional relationship between the position of the inner corner or outer corner of the eye included in the eye image and the position of the pupil or iris included in the eye image. Alternatively, the gaze tracking unit 516 may estimate and track the gaze using a machine learning model that outputs the gaze direction when an eye image is input. The gaze tracking method is not limited to the above-described method, and various other methods may be applied.

[0100] The interest estimation unit 517 estimates the product in which the user is interested (referred to as "item of interest") based on the result of tracking the user's gaze (step S53). In other words, the interest estimation unit 517 estimates the item of interest of the user based on the user's behavior history in the virtual space. [a: Example 1 of estimation of item of interest]

[0101] When the tracking result of the user's gaze indicates that the user has been directing his / her gaze at a certain product for a predetermined period of time or more, the interest estimation unit 517 may estimate that the user is interested in the certain product. In other words, the interest estimation unit 517 may estimate that the certain product is an item of interest. [b: Example 2 of Estimation of an Item of Interest]

[0102] The movement detection unit 5171 detects the movement of the user's eyes based on the eye image. When the tracking result of the user's gaze indicates that the user is directing his or her gaze at a product and the movement detection unit 5171 detects that the user's eyes have performed a predetermined movement, the interest estimation unit 517 may estimate that the corresponding product is the product of interest. The predetermined movement of the eyes may be opening the eyes to a predetermined degree or less. In other words, the predetermined movement of the eyes may be squinting, such as a wink.

[0103] The warning unit 518 determines whether the estimated item of interest is a product related to an item that the user should avoid (step S54). For example, the warning unit 518 may determine whether allergy information related to the item of interest conflicts with the allergy associated with the user, based on allergy information associated with the user. If the item of interest is a product related to an item that the user should avoid (step S54: Yes), the warning unit 518 outputs a warning (step S55). The warning unit 518 may warn that the item is related to an item that the user should avoid, for example, by outputting visual information on the display 1521. The warning unit 518 may also warn that the item is related to an item that the user should avoid, for example, by outputting auditory information from a speaker. The warning unit 518 may also warn that the item is related to an item that the user should avoid, for example, by outputting tactile information.

[0104] If the item of interest is not related to the items the user should avoid (step S54: No), the image generator 514 generates a second image including related information related to the item of interest (step S56). The related information may include information about the item of interest. The related information may also include information about other products related to the item of interest.

[0105] The display control unit 112 displays the second image (step S57). The display control unit 112 may present the second image near the item of interest as recommended information. The display control unit 112 may superimpose the second image on the first image and present it as recommended information. [5-4: Technical Effects]

[0106] The information processing system 1000c according to this disclosure can issue a warning when a user selects a product that should be avoided, thereby preventing the user from purchasing the product that should be avoided. Furthermore, the information processing system 1000c presents a second image including related information related to the product of interest, thereby further increasing the user's interest in the product of interest. [6: Sixth Embodiment]

[0107] An information processing device, an information processing system, an information processing method, and a recording medium according to a sixth embodiment will be described with reference to Figs. 16 and 17. Hereinafter, the sixth embodiment will be described using an information processing system 1000d. Note that, for the sixth embodiment, descriptions that overlap with the first to fifth embodiments will be omitted as appropriate.

[0108] In this embodiment, there is a physical store corresponding to the virtual store. The product lineup in the physical store may be the same as that in the virtual store. The store device 600 may be installed in the physical store. [6-1: Configuration of the Information Processing System 1000d]

[0109] The configuration of an information processing system 1000d as an information processing device in this embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram showing the configuration of the information processing system 1000d in this embodiment. As shown in Fig. 16, the information processing system 1000d includes a head-mounted display device 100 and an in-store device 600. [6-2: Configuration of the in-store device 600]

[0110] 16, the computing device 210 of the in-store device 600 includes a biometric authentication unit 613, an image generation unit 614, a registration information acquisition unit 315, an eye tracking unit 516, an interest estimation unit 617, and a warning unit 518, as well as a behavior history information registration unit 619, a camera image acquisition unit 620, a behavior tracking unit 621, and an inventory determination unit 622. The interest estimation unit 617 may include a first interest estimation unit 6172 and a second interest estimation unit 6173. [6-3: Information Processing Method Executed by the Information Processing System 1000d]

[0111] An information processing method executed by the information processing system 1000d will be described with reference to Fig. 17. Fig. 17 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000d. [6-3-1: Collection of information related to user interests in a virtual store]

[0112] 17A, the gaze tracking unit 516 acquires an image of the user's eyes (step S611). The gaze tracking unit 516 tracks the user's gaze using the eye image (step S612). As in the fifth embodiment, the gaze tracking unit 516 tracks the user's gaze to learn the user's interests in the virtual store. The first interest estimation unit 6172 estimates an item of interest in which the user is interested based on the tracking result of the user's gaze (step S613).

[0113] The behavior history information registration unit 619 registers information including information indicating the item of interest and information indicating the user's interests that can be inferred from the item of interest as behavior history information (step S614). The behavior history information registration unit 619 may add the behavior history information to the registered information. The behavior history information may constitute part of the registered information. [6-3-2: Collection of information related to user interests in physical stores]

[0114] As shown in FIG. 17(b), the camera image acquisition unit 620 acquires camera images (step S621). The camera image acquisition unit 620 acquires camera images from cameras installed in the physical store. The camera images capture the gaze of users visiting the physical store and the products that users pick up. The behavior tracking unit 621 tracks the user's behavior using the camera images (step S622). The behavior tracking unit 621 tracks the user's behavior in order to estimate the user's interests in the physical store.

[0115] The second interest estimation unit 6173 estimates an item of interest in which the user is interested based on the tracking results of the user's behavior (step S623). The second interest estimation unit 6173 detects, based on camera footage, that the user has picked up an item from a display shelf and that the user has returned the item to the display shelf. The second interest estimation unit 6173 may estimate that the item picked up by the user is an item of interest in which the user is interested. Furthermore, the second interest estimation unit 6173 may identify an item purchased by the user as a purchased item. Even if the user returns the item to the display shelf, the second interest estimation unit 6173 may estimate that the item is likely to be purchased by conveying its appeal.

[0116] The behavior history information registration unit 619 registers, as behavior history information, information indicating the item of interest, information indicating the user's interest that can be inferred from the item of interest, information indicating the purchased item, information indicating the user's interest that can be inferred from the purchased item, etc. (step S624). [6-3-3: Recommendations in a Virtual Store]

[0117] 17C, the image acquisition unit 111 acquires an eye image of the user (step S631). Note that the image acquisition unit 111 may acquire at least a part of the eye image acquired in step S611 as an eye image to be used for biometric authentication of the user. The biometric authentication unit 613 performs iris authentication of the user using the eye image to identify the user (step S632).

[0118] The registration information acquisition unit 315 acquires the registration information of the identified user (step S633). The registration information acquisition unit 315 may acquire the registration information of the identified user from the registration information database DB1. In this embodiment, the registration information database DB1 stores behavior history information in addition to information registered by the user.

[0119] The image generation unit 614 generates recommendation information based on the user's registration information (step S634). For example, the image generation unit 614 may identify products that the user picked up but did not purchase in a physical store based on behavioral history information, and generate a second image highlighting the appeal of the product as recommendation information. The image generation unit 614 may generate a layout image as recommendation information by superimposing the second image on a first image in which products are laid out. The image generation unit 614 may generate a recommendation image that presents the appeal of products that the user showed interest in in the physical store to the user in the virtual store. By conveying the appeal of products that the user showed interest in but did not purchase in the physical store in the virtual store, it is possible to encourage the user to purchase the products in the virtual store. The display control unit 112 presents the recommendation information generated by the image generation unit 614 (step S635).

[0120] In step S636, the information processing system 1000d performs the operation of collecting information related to the user's interests in the virtual store described with reference to FIG. 17A. That is, the information processing system 1000d collects information in the virtual store for recommendations in the real store. Also, the information processing system 1000d collects information in the real store for recommendations in the virtual store. [6-3-4: Recommendations in Real Store]

[0121] 17(d), the camera image acquisition unit 620 acquires camera image (step S641). The biometric authentication unit 613 identifies the user using the camera image (step S642). The biometric authentication unit 613 may perform face authentication using the camera image to identify the user.

[0122] The registration information acquisition unit 315 acquires the registration information of the identified user (step S643). The registration information acquisition unit 315 may acquire the registration information of the identified user from the registration information database DB1. In this embodiment, the registration information database DB1 stores behavior history information in addition to information registered by the user.

[0123] The image generating unit 614 generates recommendation information based on the user's registration information (step S644). [a: Recommendation example 1 in a physical store]

[0124] In some cases, it may be difficult for a user to decide to purchase a product in a virtual store and the user may want to pick up the product in a physical store before deciding to purchase it. When the user visits the physical store, the information processing system 1000d may guide the user to an item of interest in the virtual store.

[0125] The inventory determination unit 622 identifies an item of interest that the user is interested in in the virtual store based on the user's registration information. The inventory determination unit 622 determines whether the item of interest is in stock at the physical store that the user enters. If the item of interest is in stock at the physical store, the image generation unit 614 may generate, as recommendation information, an image showing the location of the item of interest in the physical store. The image generation unit 614 may generate an image including a map showing where the item is located in the physical store. The image generation unit 614 may generate an image including an indicator such as an arrow that guides movement within the physical store. [b: Example 2 of recommendation in a physical store]

[0126] The physical store may be an unmanned store. In an unmanned store, which is an example of a physical store, the information processing system 1000d may present recommended information for the user on signage present in the physical store to guide the user. Note that the physical store may also be a staffed store.

[0127] The display control unit 112 displays the recommended image generated by the image generation unit 314 (step S645). The display control unit 112 may present recommended information indicating the location of the product of interest in the physical store. If the product is not available in the physical store, the display control unit 112 may display a message indicating that the product is not available in the physical store.

[0128] In step S646, the information processing system 1000d performs the operation of collecting information about the user's interests in the real store, as described with reference to FIG. 17B. In other words, the information processing system 1000d collects information in the real store for recommendations in the virtual store. [6-4: Technical Effects]

[0129] The information processing system 1000d according to this disclosure can provide more beneficial services to users by linking a virtual store with a real store.

[0130] A seventh embodiment relating to an information processing device, an information processing system, an information processing method, and a recording medium will be described with reference to FIGS. 18 and 19. Hereinafter, the seventh embodiment relating to an information processing device, an information processing system, an information processing method, and a recording medium will be described using an information processing system 1000e. As described above, there are cases where it is difficult to decide to purchase a product in a virtual store, and a user wants to pick up the product in a physical store before deciding to purchase it. Therefore, this embodiment provides a form that combines a virtual store and a physical store. Note that, for the seventh embodiment, descriptions that overlap with the first to sixth embodiments will be omitted as appropriate. [7-1: Physical store in this embodiment]

[0131] The physical store in this embodiment is a place (space) that has actual products and allows users to experience the actual products. The products do not have to be on display. Therefore, the physical store in this embodiment may be a place such as a warehouse.

[0132] Furthermore, the physical store in this embodiment has a user's private space that can be used by the user. The user's private space is the user's personal space. Typically, it is a private room, and is a space separated from the private spaces of others by a partition member such as a fitting room.

[0133] In the real store in this embodiment, the user experiences shopping in this embodiment by wearing the head-mounted display device 100 in the user's private space. In this embodiment, the user can enter a virtual store in the real store. In this embodiment, the head-mounted display device 100 may be rented to the user by the real store, or may be a personal item brought in by the user. [7-2: Configuration of the information processing system 1000e]

[0134] The configuration of an information processing system 1000e as an information processing device in this embodiment will be described with reference to Fig. 18. Fig. 18 is a block diagram showing the configuration of the information processing system 1000e in this embodiment. As shown in Fig. 18, the information processing system 1000e includes a head-mounted display device 100 and a store device 700. [7-3: Configuration of the store device 700]

[0135] 18, the computing device 210 of the store device 700 includes a biometric authentication unit 613, an image generation unit 714, a registration information acquisition unit 315, an eye tracking unit 516, an interest estimation unit 717, a warning unit 518, a behavior history information registration unit 619, a camera image acquisition unit 620, a behavior tracking unit 621, and an inventory determination unit 722, as well as a transportation instruction unit 723. The interest estimation unit 717 may include a motion detection unit 7174. [7-4: Information Processing Method Executed by Information Processing System 1000e]

[0136] An information processing method executed by the information processing system 1000e will be described with reference to Fig. 19 . Fig. 19 is a flowchart showing an example of the flow of the information processing method executed by the information processing system 1000e. Note that before the flow of the information processing method executed by the information processing system 1000e illustrated in Fig. 19 , the user is identified by biometric authentication. For example, when entering a physical store, the user may be identified by face authentication based on camera images. Furthermore, when starting to use the user's private space in the physical store, the user may be identified by iris authentication based on eye images acquired by the head-mounted display device 100.

[0137] 19 , the gaze tracking unit 516 acquires an image of the user's eyes (step S71). The gaze tracking unit 516 tracks the user's gaze using the eye image (step S72). The interest estimation unit 717 estimates an item of interest in which the user is interested based on the tracking result of the user's gaze (step S73).

[0138] The stock determination unit 722 determines whether the product of interest is in stock at the physical store (step S74). If the product of interest is not in stock at the physical store (step S74: No), the process returns to step S71.

[0139] If the product of interest is in stock at the physical store (step S74: Yes), the action detection unit 7174 may ask the user to indicate whether they wish to try it. The action detection unit 7174 determines whether the user has performed an action indicating a desire to try the product of interest (step S75). If the user has not performed an action indicating a desire to try the product of interest (step S75: No), the process returns to step S71.

[0140] If the user performs an action indicating a desire to try the item of interest (step S75: Yes), the transport instruction unit 723 instructs the transport of the item of interest to the user's private space where the user is located (step S76). Examples of delivery means instructed by the transport instruction unit 723 include a conveyor belt, a drone, a delivery robot, etc. Note that a service in which delivery is performed by a human may also be implemented.

[0141] In addition, when the product is clothing, it may be necessary to put on and take off the head-mounted display device 100 in order to put on and take off the clothing. For this reason, the physical store may rent out, for example, a sports-type eyeglass-like head-mounted display device 100 that is easy to put on and take off.

[0142] The product may also be a product that allows the user to spend time comfortably in the user's private space. For example, the product may be food or drink. Below, a modified example of the operation related to inventory will be described. [a: Modification 1]

[0143] When the user visits a physical store, the display control unit 112 may present recommendation information including products that are in stock at the physical store and that the user is interested in at the virtual store. In other words, the image generation unit 714 may generate recommendation information about products that are in stock at the physical store the user is visiting. [b: Modification 2]

[0144] When the interest estimation unit 717 estimates an item of interest in which the user is interested based on the tracking results of the user's gaze (step S73), the image generation unit 714 may generate information indicating whether the corresponding item is in stock or whether the corresponding item is out of stock but can be ordered. When the corresponding item is out of stock but can be ordered and the motion detection unit 7174 detects an action that the user wishes to try, an order for the item to be ordered may be placed. Furthermore, when the corresponding item is in stock and an action that the user wishes to try is detected (step S75: Yes), the transportation instruction unit 723 may instruct delivery.

[0145] The user may choose to take the item home from the dedicated space. The user may also arrange with the delivery instruction unit 723 to have the item delivered to the user's home or a location designated by the user. [7-5: Technical Effects]

[0146] Users often want to check the feel of the actual product and how it fits to their physical body. The information processing system 1000e according to this disclosure can provide the user with the experience of searching for the product they want to try on in real life using the head-mounted display device 100 and trying out the product in their own private space. Furthermore, the information processing system 1000e can reduce the burden on both customers and store clerks.

[0147] [8: Supplementary Note] The above-described embodiment can be further described as in the following supplementary note, but is not limited to the following. [Supplementary Note 1] An information processing device comprising: an acquisition means for acquiring biometric information of a target; and a display control means for displaying a first image including products on a head-mounted display worn by the target, wherein the display control means displays the first image in which the products are laid out based on registration information of the target identified using the biometric information. [Supplementary Note 2] The information processing device according to Supplementary Note 1, comprising: an authentication means for performing biometric authentication of the target using the biometric information; and an image generation means for generating the first image based on registration information of the target identified by the biometric authentication. [Supplementary Note 3] The information processing device according to Supplementary Note 2, wherein the registration information includes information indicating at least one of preferred objects and objects to be avoided for at least one of the target and related individuals associated with the target. [Supplementary Note 4] The information processing device according to Supplementary Note 3, wherein the image generation means lays out the products to be included in the first image based on the registration information so as not to display products related to the objects to be avoided for at least one of the target and related individuals. [Supplementary Note 5] The information processing device according to Supplementary Note 2, wherein the image generation means generates the first image according to a store visited by the subject. [Supplementary Note 6] The information processing device according to Supplementary Note 3, wherein the acquisition means acquires an eye image including the eyes of the subject, and includes tracking means for tracking the gaze of the subject using the eye image. [Supplementary Note 7] The information processing device according to Supplementary Note 6, wherein the acquisition means includes estimation means for estimating an item of interest in which the subject is interested, based on the results of tracking the gaze of the subject. [Supplementary Note 8] The information processing device according to Supplementary Note 7, wherein the estimation means estimates that the subject is interested in the item of interest when the results of tracking the gaze indicate that the subject is directing their gaze at the item of interest for a predetermined period of time or more.[Supplementary Note 9] The information processing device according to Supplementary Note 7, further comprising: detection means for detecting eye movements of the subject based on the eye image, wherein the estimation means estimates that the subject is interested in the item of interest when the gaze tracking result indicates that the subject is directing their gaze toward the item of interest and the detection means detects that the subject has performed a predetermined eye movement. [Supplementary Note 10] The information processing device according to Supplementary Note 7, further comprising: warning means for issuing a warning when the item of interest is a product related to the item to be avoided of at least one of the target and the related individual. [Supplementary Note 11] The information processing device according to Supplementary Note 7, further comprising: image generation means for generating a second image based on the item of interest, the second image including information related to the item of interest, and the display control means for displaying the second image in addition to or instead of the first image. [Supplementary Note 12] The information processing device of Supplementary Note 7, comprising: a determination means for determining whether the item of interest is in a physical store visited by the subject; and if the item of interest is in the physical store, the display control means displays an image showing the location of the item of interest in the physical store. [Supplementary Note 13] The information processing device of Supplementary Note 1, comprising: a second estimation means for estimating an item of interest in which the subject has shown interest in the physical store based on camera footage from a camera installed in the physical store; and a registration means for registering information related to the item of interest as the registered information. [Supplementary Note 14] The information processing device of Supplementary Note 7, comprising: in a dedicated space of the subject in the physical store: a determination means for determining whether the item of interest is in the physical store; and a transport instruction means for instructing the item of interest to be transported to the dedicated space if the item of interest is in the physical store. [Supplementary Note 15] The information processing device according to Supplementary Note 7, wherein in a dedicated space of the target in a physical store, the image generation means generates the first image including a product in stock at the physical store, the display control means displays the first image, and the information processing device comprises a transport instruction means for instructing the transport of the product of interest estimated by the estimation means to the dedicated space. [Supplementary Note 16] The information processing device according to Supplementary Note 9, wherein the predetermined eye movement is to keep the eye opening at a predetermined level or less.[Supplementary Note 17] The information processing device according to Supplementary Note 3, wherein the image generation means highlights products related to the object to be avoided. [Supplementary Note 18] The information processing device according to Supplementary Note 1, comprising the head-mounted display. [Supplementary Note 19] An information processing system including: a first information processing device comprising: a head-mounted display worn by a subject; acquisition means for acquiring biometric information of the subject; and display control means for displaying a first image including products on the head-mounted display, wherein the display control means displays the first image in which the products are laid out based on registration information of the subject identified using the biometric information; and a second information processing device comprising: authentication means for performing biometric authentication of the subject using the biometric information; and image generation means for generating the first image based on registration information of the subject identified by the biometric authentication. [Supplementary Note 20] An information processing method executed by a computer, which acquires biometric information of a subject; and displays a first image including products on a head-mounted display worn by the subject, wherein the first image in which the products are laid out based on registration information of the subject identified using the biometric information is displayed. [Supplementary Note 21] An information processing method for acquiring biometric information of a subject, and displaying a first image including a commodity on a head-mounted display worn by the subject, wherein the first image in which the commodity is laid out based on registration information of the subject identified using the biometric information is displayed. A recording medium having recorded thereon a computer program for causing a computer to execute the information processing method.

[0148] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing systems, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0149] 10 Information processing device 11 Biometric information acquisition unit 12, 112 Display control unit 1000, 1000a, 1000b, 1000c, 1000d, 1000e Information processing system 100 Head-mounted display device 200, 300, 400, 500, 600, 700 Store device 111 Image acquisition unit 1521 Display 171, 172 Camera 213, 313, 613 Biometric authentication unit 214, 314, 414, 514, 614, 714 Image generation unit 215, 315, 415 Registration information acquisition unit DB1 Registration information database DB2 Product information database 4151 Store identification unit 4152 Target individual estimation unit 516 Gaze tracking unit 517, 617, 717 Interest estimation unit 5171 Motion detection unit 518 Warning unit 6172 First interest estimation unit 6173 Second interest estimation unit 619 Action history information registration unit 620 Camera image acquisition unit 621 Action tracking unit 622, 722 Stock determination unit 723 Transportation instruction unit 7174 Motion detection unit

Claims

1. An information processing device comprising: an acquisition means for acquiring biometric information of a subject; and a display control means for displaying a first image including a product on a head-mounted display worn by the subject, wherein the display control means displays the first image in which the product is laid out based on the registration information of the subject identified using the biometric information.

2. An information processing device as described in claim 1, comprising: an authentication means for performing biometric authentication of the target using the biometric information; and an image generation means for generating the first image based on the registration information of the target identified by the biometric authentication.

3. The information processing device according to claim 2, wherein the registration information includes information indicating at least one of desirable and avoidable items of the object and at least one of related individuals associated with the object.

4. The information processing device according to claim 3, wherein the image generation means lays out the products to be included in the first image based on the registered information so as not to display products related to the object to be avoided, which is at least one of the target and the related individual.

5. The information processing device according to claim 2, wherein the image generating means generates the first image according to a store visited by the subject.

6. The information processing device according to claim 3, wherein the acquisition means acquires an eye image including the eyes of the subject, and the information processing device is equipped with a tracking means for tracking the line of sight of the subject using the eye image.

7. The information processing device according to claim 6, further comprising an estimation means for estimating an item of interest in which the subject is interested based on the results of tracking the subject's gaze.

8. The information processing device according to claim 7, wherein the estimation means estimates that the subject is interested in the item of interest when the gaze tracking results indicate that the subject is directing their gaze at the item of interest for a predetermined period of time or longer.

9. An information processing device as described in claim 7, further comprising a detection means for detecting the eye movement of the subject based on the eye image, wherein the estimation means estimates that the subject is interested in the item of interest when the gaze tracking result indicates that the subject is directing their gaze at the item of interest and the detection means detects that the subject has made a predetermined eye movement.

10. The information processing device according to claim 7, further comprising a warning means for issuing a warning when the item of interest is a product related to at least one of the target and the related individual that should be avoided.

11. An information processing device as described in claim 7, wherein the image generation means generates a second image including information related to the item of interest based on the item of interest, and the display control means displays the second image in addition to or instead of the first image.

12. An information processing device as described in claim 7, further comprising: a determination means for determining whether the item of interest is located in a physical store visited by the subject; and if the item of interest is located in the physical store, the display control means displays an image indicating the location of the item of interest in the physical store.

13. An information processing device as described in claim 1, comprising: a second estimation means for estimating an item of interest in which the subject has shown interest in the physical store based on camera footage from a camera installed in the physical store; and a registration means for registering information regarding the item of interest as the registered information.

14. An information processing device as described in claim 7, comprising: a determination means for determining whether the item of interest is in the physical store in a dedicated space for the item in question; and a transport instruction means for instructing the transport of the item of interest to the dedicated space if the item of interest is in the physical store.

15. An information processing device as described in claim 7, wherein in a dedicated space of the target in a physical store, the image generation means generates the first image including the product in stock at the physical store, the display control means displays the first image, and the information processing device is provided with a transport instruction means for instructing the transport of the product of interest estimated by the estimation means to the dedicated space.

16. The information processing device according to claim 9, wherein the predetermined eye movement is to open the eyes to a predetermined degree or less.

17. The information processing device according to claim 3, wherein the image generating means highlights products related to the items to be avoided.

18. An information processing system including: a first information processing device comprising: a head-mounted display worn by a subject; acquisition means for acquiring biometric information of the subject; and display control means for displaying a first image including a product on the head-mounted display, wherein the display control means displays the first image in which the product is laid out based on registration information of the subject identified using the biometric information; and a second information processing device comprising: authentication means for performing biometric authentication of the subject using the biometric information; and image generation means for generating the first image based on the registration information of the subject identified by the biometric authentication.

19. An information processing method executed by a computer, which acquires biometric information of a subject and displays a first image including a product on a head-mounted display worn by the subject, wherein the first image displays the product laid out based on the registration information of the subject identified using the biometric information.

20. A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method that acquires biometric information of a subject and displays a first image including a product on a head-mounted display worn by the subject, the first image displaying the product in a layout based on the registration information of the subject identified using the biometric information.

Images