Product recommendation device, method, and program

Abstract

To accurately determine which products to recommend to the target audience. [Solution] The product recommendation device 1X mainly comprises an acquisition means 14X, an estimation means 15X, and a determination means 16X. The acquisition means 14X acquires a facial image of the subject. The estimation means 15X estimates the subject's health condition based on the facial image. The determination means 16X determines the recommended product to recommend to the subject based on the estimated health condition.

Description

[[Technical Field]]

[0001] The present disclosure relates to the technical field of a product recommendation apparatus, method and program for determining products to be recommended. [[Background Art]]

[0002] There exists a technology of recommending products suitable for a customer to the customer. For example, Patent Document 1 discloses a product recommendation system that acquires purchase history data associated with a customer based on customer data, and determines products to be recommended to the customer by referring to the purchase history data. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2024-071165 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] A prospective purchaser who is considering purchasing a product at a store can use recommended products matching their current condition as a reference when deciding which product to purchase. On the other hand, there is a problem that recommended products differ depending on the person, and even for the same person, recommended products differ depending on the person's current condition and other factors.

[0005] One object of the present disclosure, in view of the above-mentioned problem, is to provide a product recommendation apparatus, method, and program capable of accurately determining products to be recommended to a subject. [[Means for Solving the Problem]]

[0006] One aspect of the product recommendation apparatus is: acquisition means for acquiring a face image of the subject; and estimation means for estimating the health condition of the subject based on the face image; A decision-making means for determining recommended products to be recommended to the subject based on the estimated health status, It is a product recommendation device that has the following features.

[0007] One aspect of the method is: Computers We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. Based on the estimated health status, a recommended product is determined for the subject. It is a method.

[0008] One aspect of the program is: We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. This program causes a computer to perform a process to determine recommended products to the subject based on the estimated results of their health condition. [Effects of the Invention]

[0009] One example of the benefits of this disclosure is that it enables accurate determination of which products to recommend to the target audience. [Brief explanation of the drawing]

[0010] [Figure 1] This outlines the general configuration of the product recommendation system. [Figure 2] This shows the hardware configuration of the product recommendation device. [Figure 3] This is the first functional block diagram of the product recommendation system. [Figure 4] This is an example of how information about recommended products is displayed. [Figure 5] This is an example of a flowchart illustrating the steps involved in the product recommendation process. [Figure 6] This is the second functional block diagram of the product recommendation system. [Figure 7] This is an example of how information about recommended products is displayed. [Figure 8] This is the third functional block diagram of the product recommendation system. [Figure 9] It is a fourth functional block diagram of the product recommendation device. [Figure 10] It is a fifth functional block diagram of the product recommendation device. [Figure 11] This is an example of a flowchart executed by the product recommendation device. MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, embodiments of a product recommendation apparatus, method, and program will be described with reference to the drawings.

[0012] <First Embodiment> Figure 1 shows a schematic configuration of a product recommendation system 100. The product recommendation system 100 estimates a subject's health condition based on a face image of the subject, determines a product to recommend to the subject based on the estimated health condition, and notifies the subject of the determined product. The product recommendation system 100 may be a system installed at an entrance or the like of a store that sells products, or may be a mobile terminal such as a smartphone or tablet terminal used by the subject, or a personal computer.

[0013] The product recommendation system 100 mainly includes a product recommendation device 1, an input device 2, an output device 3, a storage device 4, and a measurement device 5 including a camera (photographing device) 51. A subject is a person who is the subject of the camera 51 and receives product recommendations. Hereinafter, a face video is a time-series image capturing the face of the subject. Note that the face video may be configured from a single image.

[0014] The product recommendation device 1 determines a product to recommend to the subject (also referred to as a "recommended product") based on the subject's health condition estimated from the subject's face video generated by the camera 51, and notifies the subject of information related to the determined recommended product. The product recommendation device 1 performs data communication with the input device 2, the output device 3, the storage device 4, and the measurement device 5 via a communication network, or by direct wireless or wired communication.

[0015] Input device 2 is an interface that accepts user input (external input) from the target person. Input device 2 may be various user input interfaces such as a touch panel, buttons, keyboard, mouse, or voice input device. Input device 2 supplies the input signal generated based on the user input to product recommendation device 1.

[0016] Output device 3 outputs predetermined information based on the output signal supplied from product recommendation device 1. In this case, the output signal includes at least one of a display signal or an audio signal. Output device 3 then displays the information based on the display signal supplied from product recommendation device 1 and outputs the information as audio based on the audio signal supplied from product recommendation device 1. Output device 3 includes, for example, at least one of a display device such as a display or projector and an audio output device such as a speaker. When the product recommendation system 100 is installed in a store, output device 3 is, for example, a display that presents recommended products to a target customer who has entered the store.

[0017] The storage device 4 is a memory that stores various information necessary for determining recommended products and presenting information about those recommended products. For example, the storage device 4 stores product information about products that are candidates for recommended products. The product information is a database that associates information such as the price, thumbnail, and other product attributes (such as the nutrients the product contains) with each product identification information. The storage device 4 may be an external storage device such as a hard disk connected to or built into the product recommendation device 1, or it may be a storage medium such as flash memory. The storage device 4 may also be a server device that communicates data with the product recommendation device 1. Furthermore, the storage device 4 may be composed of multiple devices.

[0018] The measuring device 5 is one or more sensors, including a camera 51. If the product recommendation system 100 is installed in a store, the camera 51 is, for example, a camera that photographs customers entering the store. In another example, if a mobile device carried by a subject functions as the product recommendation system 100, the camera 51 is a camera built into the mobile device.

[0019] The configuration of the product recommendation system 100 shown in Figure 1 is an example, and various modifications may be made to this configuration. As described above, the product recommendation device 1, input device 2, output device 3, storage device 4, and measuring device 5 may all be implemented as a single device such as a smartphone or tablet terminal.

[0020] Figure 2 shows an example of the hardware configuration of product recommendation device 1. Product recommendation device 1 includes a processor 11, memory 12, and interface 13 as hardware. The processor 11, memory 12, and interface 13 are connected via a data bus 19.

[0021] The processor 11 functions as a controller (arithmetic unit) that controls the entire product recommendation device 1 by executing programs stored in memory 12. The processor 11 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may consist of multiple processors. The processor 11 is an example of a computer.

[0022] Memory 12 is composed of various volatile and non-volatile memories, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores the program executed by the product recommendation device 1. If the product recommendation device 1 is a mobile terminal, memory 12 may have an application installed to implement the processing performed by the product recommendation device 1 in this disclosure. In this case, the product recommendation device 1 executes the processing in this disclosure based on the installed application.

[0023] Furthermore, some of the information stored in memory 12 may be stored in an external storage device such as a storage device 4 that can communicate with the product recommendation device 1, or it may be stored in a storage medium that can be attached to or removed from the product recommendation device 1. In addition, memory 12 may function as a storage device 4 and store the information that storage device 4 stores instead.

[0024] Interface 13 is an interface for electrically connecting the product recommendation device 1 with other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data with other devices, or they may be hardware interfaces for connecting with other devices via cables, etc. Interface 13 may also include a user interface, and may perform interface operations between a device that functions as a user interface and the product recommendation device 1.

[0025] Figure 3 shows an example of the functional blocks of the product recommendation device 1 related to the learning of the internal state estimation model. Functionally, the processor 11 of the product recommendation device 1 includes a face image acquisition unit 14, a health state estimation unit 15, a recommended product determination unit 16, and a notification unit 17. In Figure 3, blocks where data is exchanged are connected by solid lines, but the combinations of blocks where data is exchanged are not limited to those shown. The same applies to the diagrams of other functional blocks described later.

[0026] The face image acquisition unit 14 acquires the subject's face image supplied from the camera 51 via the interface 13. In this case, for example, the face image acquisition unit 14 performs a process to detect a person's face from the time-series images supplied from the camera 51 using any face detection technology, and when a person's face is detected, it extracts a predetermined number of time-series images in which the face was detected as a face image. The predetermined number mentioned above is, for example, determined according to the input format of the model described later, which takes face images as input. The face image acquisition unit 14 then supplies the extracted face image to the health status estimation unit 15.

[0027] The health status estimation unit 15 estimates the health status of the subject based on the facial image acquired by the facial image acquisition unit 14. Here, examples of the health status estimation results include arbitrary index values ​​of vital signs estimated by vital status estimation, and index values ​​estimated based on vital signs. Examples of vital signs include heart rate, heart rate, heart rate variability, SpO2, blood pressure, pulse variability, blood glucose level, cholesterol level, etc. Examples of index values ​​estimated based on vital signs include stress level, fatigue level, drowsiness level, BMI, dehydration level, and cognitive function index values. When estimating health status from facial images, for example, the techniques described in WO2019 / 123569, JP 2024-008099, etc. may be used.

[0028] The health status estimation unit 15 estimates the health status of a subject from facial images using, for example, an arbitrary model (also called the "health status estimation model") that estimates the health status of a subject from one or more facial images. The health status estimation model is, for example, a machine learning model (including statistical models, the same applies hereinafter) and has an arbitrary architecture adopted in machine learning such as neural networks and support vector machines. The health status estimation model has been pre-programmed with machine learning to output an estimation result regarding the health status of a subject when a predetermined number of facial images are input. In other words, the health status estimation model is a model that has learned the relationship between a predetermined number of facial images and the health status of a subject through machine learning. The storage device 4 or memory 12, etc., stores the machine-learned parameters of the health status estimation model. For example, if the health status estimation model described above is a model based on a neural network such as a convolutional neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter are pre-stored in the storage device 4 or memory 12, etc. The health status estimation unit 15 constructs a health status estimation model by referring to machine learning parameters that have been pre-stored in the storage device 4 or memory 12, and obtains the health status estimation result output by the health status estimation model by inputting the face video acquired from the face image acquisition unit 14 into the health status estimation model.

[0029] For example, a health status estimation model may estimate heart rate from facial images, or the degree of respiration from facial images, or the SpO2 from facial images. In other examples, a health status estimation model may estimate blood pressure from facial images, or blood glucose levels from facial images, or cholesterol levels from facial images, or the degree of edema from facial images. In yet another example, a health status estimation model may estimate the state of wrinkles and the risk of physical disease based on the state of wrinkles from facial images, or a frail state correlated with the risk of physical and brain diseases from facial images, or the degree of sleepiness from facial images. In yet another example, a health status estimation model may estimate concentration levels from facial images, or the degree of stress from facial images, or the balance between the sympathetic nervous system (LF) and the parasympathetic nervous system (HF) (LF / HF) from facial images, or the degree of cognitive function from facial images.

[0030] The health status estimation unit 15 may estimate the subject's health status using multiple health status estimation models that estimate the health status of different indicators. For example, the health status estimation unit 15 uses health status estimation models that estimate drowsiness, concentration, and cognitive function, respectively, to obtain estimation results for drowsiness, concentration, and cognitive function. The health status estimation unit 15 then determines that if the subject is not drowsy but is not concentrating, their brain and mental health is worse; if cognitive function is high but drowsiness is severe, their mental or physical health is worse; and if cognitive function is high but concentration is weak, their mental health is worse. In this way, the health status estimation unit 15 can estimate the health status of multiple indicators and, by combining these, can identify worse health conditions or identify health conditions that require attention. In the latter example, if drowsiness is severe, all aspects of mental, physical, and brain health could be considered, but if cognitive function is high, the issue can be narrowed down to either mental or physical health.

[0031] Furthermore, the health status estimation unit 15 may estimate the subject's future health status. In this case, the health status estimation unit 15 estimates the future health status, for example, by linear regression, based on the time-series estimation results (i.e., time-series information) obtained by continuously monitoring the subject's current health status. The health status estimation unit 15 may also estimate the future health status using LSTM (Long Short Term Memory) or the like, with time-series facial images as input.

[0032] The recommended product determination unit 16 determines recommended products based on the health status estimation results supplied by the health status estimation unit 15. In this case, the recommended product determination unit 16 may determine recommended products from the health status estimation results using, for example, a machine learning model (also called the "recommended product determination model") that has been trained to learn the relationship between health status and products to be recommended. The recommended product determination model is a model that has been trained to output identification information of products that are suitable for a person's health status when the estimated results of a person's health status are input. The recommended product determination model is, for example, a machine learning model (including statistical models, the same applies hereinafter) and has any architecture adopted in machine learning such as neural networks and support vector machines. The storage device 4 or memory 12, etc., stores the machine learning parameters of the recommended product determination model. For example, if the recommended product determination model described above is a model based on a neural network such as a convolutional neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter are stored in advance in the storage device 4 or memory 12, etc. The recommended product determination unit 16 constructs a recommended product determination model by referring to parameters, and inputs the health status estimation results obtained from the health status estimation unit 15 into the recommended product determination model. The unit then obtains the product identification information output by the recommended product determination model as the recommended product identification information.

[0033] The recommended product determination model may also be a lookup table that associates the estimated health status with recommended products. In this case, the recommended product determination unit 16 determines the recommended products by referring to the estimated health status and the lookup table.

[0034] The notification unit 17 notifies the target person of information regarding the recommended product determined by the recommended product determination unit 16 by displaying and / or outputting it as sound via the output device 3. In this case, the notification unit 17 refers to the product information of the recommended product stored in the storage device 4, etc., based on the identification information of the recommended product supplied from the recommended product determination unit 16. The notification unit 17 then generates output information regarding the recommended product and supplies the generated output information to the output device 3, causing the output device 3 to display and / or output the information regarding the recommended product as sound. In this case, the notification unit 17 may also output the estimated health status result generated by the health status estimation unit 15 to the output device 3 along with the information regarding the recommended product.

[0035] The facial image acquisition unit 14, health status estimation unit 15, recommended product determination unit 16, and notification unit 17 can be implemented, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to implement each component. At least a portion of these components may be implemented not only by software programs, but also by any combination of hardware, firmware, and software. At least a portion of these components may also be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or microcontroller. In this case, the program composed of the above components may be implemented using this integrated circuit. At least a portion of each component may also be implemented using an ASSP (Application Specific Standard Produce), ASIC (Application Specific Integrated Circuit), or quantum processor (quantum computer control chip). Thus, each component may be implemented using various hardware. The same applies to other embodiments described later. Furthermore, each of these components may be implemented by the collaboration of multiple computers, for example, using cloud computing technology.

[0036] Figure 4 shows an example of the display of information regarding recommended products that the notification unit 17 displays on the output device 3. Based on the health status estimation result supplied from the health status estimation unit 15 and the recommended product identification information supplied from the recommended product determination unit 16, the notification unit 17 generates a display signal to supply to the output device 3. By supplying the generated display signal to the output device 3, the screen shown in Figure 4 is displayed on the output device 3.

[0037] In the example shown in Figure 4, the notification unit 17 displays text information based on the estimated health status, "Your current health status is XXX." (where "XXX" is a string representing the estimated health status), along with information on recommended products, including products A, B, and C (thumbnail images in this case). For example, if the product recommendation system 100 is installed in a store, the product recommendation device 1 will present this information via the output device 3 to customers entering the store. In this way, the product recommendation device 1 can present products suitable for the target person's current health status and promote product purchase. Furthermore, even if the product recommendation system 100 is a terminal such as a smartphone, the product recommendation device 1 can encourage the target person to make a purchase in the store by displaying information as shown in Figure 4.

[0038] Recommended products are not limited to items sold in stores (including miscellaneous goods), but may also include experiential products such as travel plans.

[0039] Figure 5 is an example flowchart showing the steps of the product recommendation process performed by the product recommendation device 1. The product recommendation device 1 repeatedly executes the steps in the flowchart shown in Figure 5.

[0040] First, the product recommendation device 1 determines whether or not a facial image has been acquired based on the time-series images supplied from the camera 51 (step S11). In this case, for example, if the product recommendation device 1 determines that it has obtained the number of time-series facial images of the same person necessary for estimating health status, it determines that a facial image has been acquired. If the product recommendation device 1 determines that a facial image has not been acquired (step S11; No), it terminates the flowchart process.

[0041] If the product recommendation device 1 determines that it has acquired a facial image (step S11; Yes), it estimates the health status of the subject in the facial image based on the acquired facial image (step S12). In this case, for example, the product recommendation device 1 inputs the facial image into a health status estimation model that has been pre-machine-learned, and obtains the health status inference result output by the health status estimation model.

[0042] Then, the product recommendation device 1 determines recommended products for the subject based on the estimated health status (step S13). In this case, the product recommendation device 1 determines recommended products from the estimated health status using, for example, a recommended product determination model. Then, the product recommendation device 1 outputs information about the recommended products determined in step S13 via the output device 3 (step S14).

[0043] Furthermore, the product recommendation device 1 may, for example, link with a POS system in a store equipped with the product recommendation system 100 and acquire purchase data indicating the products purchased by the target person after recommended products are presented. In this case, the product recommendation device 1 stores the estimated results of the target person's health status and the target person's purchase data in association and uses them for machine learning of the recommended product decision model. For example, the product recommendation device 1 may update the parameters of the recommended product decision model by using the estimated results of the target person's health status as input data to the recommended product decision model and the target person's purchase data as the correct data that the recommended product decision model should output. This can improve the accuracy of the recommended product decision model. The product recommendation device 1 may also transmit information linking the estimated results of the target person's health status and the target person's purchase data to a server device that performs machine learning of the recommended product decision model. In this case, the server device performs machine learning of the recommended product decision model using the pair of the estimated results of the target person's health status and the target person's purchase data as training data. The product recommendation device 1 may also determine recommended products using a recommended product decision model trained with purchase data from people other than the target person (or including people other than the target person) and health status data (estimated results). Furthermore, a general-purpose product recommendation model, which is trained using purchase data and health status data from individuals other than the target person (or including individuals other than the target person), may be further trained (i.e., fine-tuned) by adding the estimated health status and purchase data of the target person as training data. In this case, the product recommendation device 1 determines the recommended product using the product recommendation model obtained by fine-tuning the general-purpose product recommendation model.

[0044] Furthermore, even if health status data is unavailable, various purchase data from different buyers is available, and it is common practice to already analyze purchase trends based on buyer attributes or the time of purchase (season) using such purchase data. Considering the above, even if there is no health status data to train the aforementioned product recommendation model, a general-purpose model (general-purpose product recommendation model) that determines recommended products based on buyer attributes and time of year (seasonal information), generated based on past purchase data, may be used. In this case, product recommendation device 1 may output both a recommendation result for a product estimated based on attributes and time of year, and a recommendation result based on the health status at that time, using the general-purpose model described above, for first-time customers (i.e., customers for whom health status data has not been obtained). Thus, product recommendation device 1 may use a general-purpose model when health status data is scarce. This makes it possible to further improve sales.

[0045] Furthermore, product recommendations by the Product Recommendation System 100 introduce products that contain some of the nutrients or ingredients generally considered effective, and do not necessarily have the purpose of recommending the consumption or use of the product in question. In addition, product recommendations by the Product Recommendation System 100 are solely for introducing products available in the store and are not intended for the purpose of providing nutritional guidance to customers, improving customers' health conditions, preventing diseases, identifying symptoms, etc., and do not have any medical purpose whatsoever.

[0046] <Second Embodiment> In the second embodiment, the product recommendation device 1 receives a designation of a desired state, which is the state desired by the subject, and determines recommended products based on the estimated health status and the designated desired state. Thus, the product recommendation device 1 presents the subject with recommended products that are suitable for them. The configuration of the product recommendation system 100 and the hardware configuration of the product recommendation device 1 in the second embodiment are the same as those shown in Figures 1 and 2, respectively. Elements of the second embodiment that are the same as those of the first embodiment are appropriately denoted by the same reference numerals as those of the first embodiment, and their descriptions are omitted.

[0047] Figure 6 shows an example of a functional block of the processor 11. Functionally, the processor 11 includes a face image acquisition unit 14, a health status estimation unit 15, a recommended product determination unit 16, a notification unit 17, and a desired state identification unit 18. The face image acquisition unit 14, the health status estimation unit 15, and the notification unit 17 perform the processes described using Figure 3.

[0048] The Desired State Identification Unit 18 receives an external input (user input) from the input device 2 that specifies the desired state, which is the state desired by the subject. Based on the input signal supplied from the input device 2, the Desired State Identification Unit 18 identifies the specified desired state. The desired state is the internal state desired by the subject. In other words, the desired state is a relative state based on the current internal state (emotions), and is the state desired by the subject. Examples of desired states include "I want to calm down," "I want to focus," "I want to improve my concentration," and "I want to refresh myself." The Desired State Identification Unit 18 then supplies information indicating the specified desired state to the Recommended Product Determination Unit 16.

[0049] For example, multiple candidates (options) for the desired state specified by the user are stored in the storage device 4 or memory 12, etc., and the desired state identification unit 18 presents these multiple candidates for the desired state in a selectable format via the output device 3. The desired state identification unit 18 then identifies the candidate for the desired state selected by the input device 2 from the presented candidates as the specified desired state.

[0050] In the first example, when the health status estimation result is obtained by the health status estimation unit 15, the desired state identification unit 18 notifies the subject of the multiple candidate desired states mentioned above along with the health status estimation result via the output device 3 and accepts the designation of the desired state. In the second example, the desired state identification unit 18 may notify the subject of the multiple candidate desired states mentioned above via the output device 3 and accept the designation of the desired state before the health status estimation result is obtained by the health status estimation unit 15 (for example, before the facial image is generated). In the second example, for example, the product recommendation device 1 is a mobile terminal used by the subject, and it accepts the designation of the desired state in advance and stores the designated desired state as the subject's setting information.

[0051] The desired state identification unit 18 may also select a group of candidate desired states to present to the subject based on the health state estimation results from the health state estimation unit 15. In this case, for example, relational information such as a lookup table showing the relationship between the health state estimation results and the group of candidate desired states to be presented is stored in the storage device 4 or memory 12, etc.

[0052] The recommended product determination unit 16 determines a recommended product based on the estimated health status supplied by the health status estimation unit 15 and the desired status identified by the desired status identification unit 18. In this case, the recommended product determination unit 16 may determine a recommended product from the estimated health status using a recommended product determination model, which is a machine learning model that has learned the relationship between the health status, desired status, and the products to be recommended. In this case, the recommended product determination model is a model that has been machine-learned to output identification information for products that are suitable for a person's health status and desired status when the estimated health status and desired status of that person are input. The recommended product determination model is, for example, a machine learning model and has any architecture adopted in machine learning, such as a neural network or a support vector machine. The storage device 4 or memory 12, etc., stores the machine-learned parameters of the recommended product determination model. The recommended product determination unit 16 constructs the recommended product determination model by referring to the parameters and inputs the estimated health status and identified desired status obtained from the health status estimation unit 15 into the recommended product determination model, thereby obtaining the product identification information output by the recommended product determination model as the identification information for the recommended product. The recommended product selection model may be a statistical model, or it may be a lookup table that associates estimated health status and desired status with recommended products.

[0053] In a preferred example, the recommended product determination unit 16 may identify the nutrients necessary for the subject based on the estimated health status supplied by the health status estimation unit 15 and the desired status identified by the desired status identification unit 18, and then determine a food product containing the identified nutrients as a recommended product.

[0054] In this case, the recommended product determination model is a machine learning model that has learned the relationship between pairs of health status and desired status and the nutrients that should be consumed according to these statuses. The machine learning parameters of the recommended product determination model are stored in the storage device 4 or memory 12, etc. The recommended product determination unit 16 constructs the recommended product determination model by referring to the parameters and identifies the nutrients that the recommended product determination model will output by inputting the estimated health status results obtained from the health status estimation unit 15 and the identified desired status into the recommended product determination model. Next, the recommended product determination unit 16 identifies recommended products that contain the identified nutrients. In this case, for example, the storage device 4, etc. stores a database of product information where information on the nutrients contained in each product is linked, and the recommended product determination unit 16 refers to the database of product information and identifies products containing the identified nutrients as recommended products. In this way, the recommended product determination unit 16 can take into account the subject's health status and present the subject with recommended products that contain the nutrients that should be consumed to get closer to their desired status.

[0055] Figure 7 shows an example of how the notification unit 17 displays information about recommended products on the output device 3. In Figure 7, the notification unit 17 determines a recommended product after the user has selected a desired state from several candidate states, and displays information about the determined recommended product on the output device 3.

[0056] First, the notification unit 17 displays the estimated health status supplied by the health status estimation unit 15, and displays the desired state selection buttons 41 to 43 corresponding to the desired state candidates, prompting the user to select a desired state. Here, the desired state selection button 41 is a user interface for specifying a state of greater calmness as the desired state. The desired state selection button 42 is a user interface for specifying a state of greater focus as the desired state. The desired state selection button 43 is a user interface for specifying a state of greater concentration as the desired state.

[0057] In the example shown in Figure 7, when the desired state selection button 43 is selected by the user, the notification unit 17 highlights the desired state selection button 43. Furthermore, the notification unit 17 determines that the desired state corresponding to the selected desired state selection button 43 has been specified, and based on the specified desired state and the estimated health status, it determines recommended products. Here, the notification unit 17 displays information (thumbnail images in this case) of the recommended products, including products AA, BB, and CC. For example, if the product recommendation system 100 is installed in a store, the product recommendation device 1 presents this information via the output device 3 to customers entering the store. In this way, the product recommendation device 1 can present products suitable for the target person's current health status and promote product purchase. Also, even if the product recommendation system 100 is a terminal such as a smartphone, the product recommendation device 1 can promote purchases in the store by displaying information as shown in Figure 7.

[0058] <Third Embodiment> In the third embodiment, the product recommendation device 1 estimates the internal state of the subject other than the health state estimated by the health state estimation unit 15, and determines recommended products based on the estimated internal state and the estimated health state. As a result, the product recommendation device 1 presents the subject with recommended products that are more suitable for them.

[0059] The configuration of the product recommendation system 100 and the hardware configuration of the product recommendation device 1 in the second embodiment are the same as those shown in Figures 1 and 2, respectively. Elements of the second embodiment that are the same as those of the first embodiment are appropriately denoted by the same reference numerals as those of the first embodiment, and their descriptions are omitted.

[0060] Figure 8 shows an example of a functional block of the processor 11. Functionally, the processor 11 includes a face image acquisition unit 14, a health state estimation unit 15, a recommended product determination unit 16, a notification unit 17, and an internal state estimation unit 18A. The face image acquisition unit 14, the health state estimation unit 15, and the notification unit 17 perform the processes described using Figure 3.

[0061] The internal state estimation unit 18A estimates the subject's internal state based on the facial image supplied from the facial image acquisition unit 14. Here, the internal state is, for example, emotions, and the internal state estimation unit 18A generates estimation results for indicators different from the health state estimated by the health state estimation unit 15. For example, the health state estimated by the health state estimation unit 15 is the subject's state other than their internal state (or emotions). The internal state estimation unit 18A supplies the estimation results of the subject's internal state to the recommended product determination unit 16.

[0062] The internal state estimation unit 18A may estimate the internal state of a subject from a facial image by using a machine learning model (also called the "internal state estimation model") which has been trained to learn the relationship between a predetermined number of facial images of the subject and the subject's internal state. In this case, the internal state estimation model is a model that has been trained to output an estimation result of the internal state of the subject who is the subject of the image sequence when a predetermined number of facial images are input. The internal state estimation model is, for example, a machine learning model and has any architecture adopted in machine learning, such as a neural network or a support vector machine. The storage device 4 or memory 12 stores the trained parameters of the internal state estimation model. The recommended product determination unit 16 constructs the internal state estimation model by referring to the parameters. The recommended product determination unit 16 then inputs the facial image into the internal state estimation model to obtain the internal state estimation result output by the internal state estimation model.

[0063] The recommended product determination unit 16 determines recommended products based on the estimated health status supplied by the health status estimation unit 15 and the estimated internal state supplied by the internal state estimation unit 18A. In this case, the recommended product determination unit 16 may determine recommended products from the estimated health status and internal state using a recommended product determination model, which is a machine learning model that has learned the relationship between health status and internal state and products that should be recommended to a person in these states. In this case, the recommended product determination model is a model that has been machine-learned to output identification information of products suitable for a person with the estimated health status and internal state when the estimated health status and internal state are input. The recommended product determination model is, for example, a machine learning model and has any architecture adopted in machine learning, such as a neural network or a support vector machine. The storage device 4 or memory 12, etc., stores the machine-learned parameters of the recommended product determination model. The recommended product determination unit 16 constructs a recommended product determination model by referring to parameters, and inputs the estimated health status and internal state results obtained from the health status estimation unit 15 into the recommended product determination model. The product identification information output by the recommended product determination model is then obtained as the recommended product identification information. The recommended product determination model may be a statistical model, or it may be a lookup table that associates the estimated health status and internal state results with the recommended products.

[0064] Furthermore, the recommended product determination unit 16 may identify the nutrients necessary for the subject based on the health status estimation results supplied from the health status estimation unit 15 and the internal state estimation results supplied from the internal state estimation unit 18A, and determine foods containing the identified nutrients as recommended products. In this case, for example, the storage device 4 stores a database of product information linked to the nutrients contained in each product, and the recommended product determination unit 16 refers to the product information database and identifies products containing the identified nutrients as recommended products.

[0065] Thus, according to the third embodiment, even if the subject's desired state cannot be obtained as in the second embodiment, it is possible to determine recommended products that reflect the estimated internal state of the subject.

[0066] <Fourth Embodiment> In the fourth embodiment, the product recommendation device 1 acquires relevant information other than the subject's health status that influences the subject's decision on which products to purchase (i.e., related to the products to be purchased), and determines recommended products based on the relevant information and the estimated results of the health status. As a result, the product recommendation device 1 presents the subject with the recommended products determined considering the relevant information.

[0067] The configuration of the product recommendation system 100 and the hardware configuration of the product recommendation device 1 in the fourth embodiment are the same as those shown in Figures 1 and 2, respectively. Elements of the fourth embodiment that are the same as those of the first embodiment are appropriately denoted by the same reference numerals as those of the first embodiment, and their descriptions are omitted.

[0068] Figure 9 shows an example of a functional block of the processor 11. Functionally, the processor 11 includes a face image acquisition unit 14, a health status estimation unit 15, a recommended product determination unit 16, a notification unit 17, and a related information acquisition unit 18B. The face image acquisition unit 14, the health status estimation unit 15, and the notification unit 17 perform the processes described using Figure 3.

[0069] The related information acquisition unit 18B acquires related information and supplies the acquired related information to the recommended product determination unit 16. Here, examples of related information include the subject's attribute information, the date and time information when the facial image was acquired, and purchase data, which is information showing the history of products the subject has purchased in the past. Attribute information is information that represents arbitrary attributes of the subject, such as gender and age. If the related information is attribute information, the related information acquisition unit 18B may, for example, identify the subject's attributes by analyzing the facial image acquired by the facial image acquisition unit 14 using any facial recognition technology. In another example, the related information acquisition unit 18B may acquire the subject's attribute information by referring to user information that links the subject's identification information (user ID) identified using facial recognition technology, etc., from the facial image, etc., with the subject's attribute information. Also, if the related information is date and time information, for example, the related information acquisition unit 18B acquires the date and time information that has been added as metadata to the facial image acquired by the facial image acquisition unit 14. Furthermore, if the related information is purchase data, the related information acquisition unit 18B acquires purchase data linked to the identification information (user ID) of the subject identified using facial recognition technology or the like from facial video footage. The user information and purchase data described above are stored, for example, in the storage device 4.

[0070] The recommended product determination unit 16 determines recommended products based on the health status estimation results supplied by the health status estimation unit 15 and the related information acquired by the related information acquisition unit 18B. In this case, the recommended product determination unit 16 may determine recommended products from the health status estimation results using a recommended product determination model, which is a machine learning model that has learned the relationship between health status, related information, and products to be recommended. In this case, the recommended product determination model is a model that has been machine-learned to output identification information of products selected as products to be recommended to a person with the estimated health status, taking the related information into consideration, when the health status estimation results and related information are input. The recommended product determination model is, for example, a machine learning model and has any architecture adopted in machine learning, such as a neural network or a support vector machine. The storage device 4 or memory 12, etc., stores the machine-learned parameters of the recommended product determination model. The recommended product determination unit 16 constructs the recommended product determination model by referring to the parameters and inputs the health status estimation results and related information into the recommended product determination model to acquire the product identification information output by the recommended product determination model as the identification information of the recommended product. The recommended product selection model may be a statistical model, or it may be a lookup table that associates estimated health status and desired status with recommended products.

[0071] For example, if the relevant information is date and time information, the season, weather, and time of day can be identified from the date and time information, so the recommended product determination unit 16 can use the relevant information to determine recommended products that are appropriate to the external environment. In another example, if the relevant information is purchase data, the target person's product purchasing tendencies can be identified, so the recommended product determination unit 16 can use the relevant information to determine recommended products that take into account the target person's purchasing tendencies. Furthermore, even if the relevant information is attribute information, product purchasing tendencies can be identified for each attribute, so the recommended product determination unit 16 can use the relevant information to determine recommended products that take into account purchasing tendencies according to the attributes.

[0072] <Fifth Embodiment> Figure 10 is a block diagram of the product recommendation device 1X. The product recommendation device 1X mainly comprises an acquisition means 14X, an estimation means 15X, and a determination means 16X. The product recommendation device 1X may be composed of multiple devices.

[0073] The acquisition means 14X acquires a facial image of the subject. The acquisition means 14X can be, for example, the facial image acquisition unit 14 in the first to fourth embodiments.

[0074] The estimation means 15X estimates the subject's health status based on facial images. The estimation means 15X can be, for example, the health status estimation unit 15 in the first to fourth embodiments.

[0075] The determination means 16X determines recommended products to recommend to the subject based on the estimated health status. The determination means 16X can be the recommended product determination unit 16 in the first to fourth embodiments.

[0076] Figure 11 shows an example of a flowchart executed by the product recommendation device 1X. The acquisition means 14X acquires a facial image of the subject (step S21). The estimation means 15X estimates the subject's health status based on the facial image (step S22). The decision means 16X determines the recommended product to recommend to the subject based on the estimated health status (step S23).

[0077] According to the fifth embodiment, the product recommendation device 1X can determine recommended products that are suitable for the target person.

[0078] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a computer, such as a processor. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage mediums (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage mediums (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer using various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0079] Furthermore, some or all of the above embodiments (including modified versions, hereinafter the same) may also be described as follows, but are not limited to the following. Also, some or all of the configurations described in Appendices 2 to 8, which are dependent on Appendice 1 above, may also be dependent on Appendices 9 and 10 in the same way as Appendices 2 to 8. Moreover, not limited to the devices, methods, and storage media described in the appendices, some or all of the configurations described in the appendices may also be dependent on methods, various hardware, software, various recording means for recording software (including storage media), or systems, without departing from the above embodiments.

[0080] [Note 1] A means of acquiring facial images of the subject, An estimation means for estimating the health status of the subject based on the aforementioned facial image, A decision-making means for determining recommended products to be recommended to the subject based on the estimated health status, A product recommendation device having the following features. [Note 2] The system further includes a means for identifying a desired state, which is the state of the subject that the subject desires. The product recommendation device described in Appendix 1 determines the recommended product based on the specified desired state and the estimated result of the health state. [Note 3] The product recommendation device described in Appendix 2, wherein the determination means identifies the nutrients necessary for the subject based on the identified desired state and the estimated result of the health state, and determines the recommended product containing the identified nutrients. [Note 4] The estimation means further estimates the inner state of the subject based on the facial image, The product recommendation device described in Appendix 1, wherein the determination means determines the recommended product based on the estimated health status and the estimated internal state. [Note 5] The acquisition means further acquires, as relevant information related to the purchase of the subject, at least one of the subject's attribute information, the date and time information of the facial image, or purchase data showing the subject's past purchase history. The product recommendation device described in Appendix 1 determines the recommended product based on the estimated health status and the related information. [Note 6] The product recommendation device described in Appendix 1, further comprising a notification means for displaying or outputting audio information regarding the aforementioned recommended products. [Note 7] The product recommendation device described in Appendix 2, wherein the identifying means displays a plurality of candidates for the desired state together with the estimated result of the health state, and identifies the candidate selected from the plurality of candidates as the desired state. [Note 8] The aforementioned determination means determines the recommended product based on a machine learning model that has learned the relationship between health status and products suitable for a person with that health status, and the estimated results of the health status. The product recommendation device according to Appendix 1, further comprising a learning means for acquiring purchase data showing the products purchased by the subject after the estimation of the health status, and for performing machine learning on the machine learning model based on the estimated health status and the purchase data. [Note 9] Computers We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. Based on the estimated health status, a recommended product is determined for the subject. method. [Note 10] We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. A program that causes a computer to perform a process to determine recommended products to recommend to the subject based on the estimated results of the health status. [Note 11] A storage medium containing the program described in Appendix 10.

[0081] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of symbols]

[0082] 1, 1X product recommendation device 2 Input devices 3. Output device 4 Storage device 5. Measuring device 11 processors 12 memory 13 Interfaces 51 Camera 100 Product Recommendation System

Claims

1. A means of acquiring facial images of the subject, An estimation means for estimating the health status of the subject based on the aforementioned facial image, A decision-making means for determining recommended products to be recommended to the subject based on the estimated health status, A product recommendation device having the following features.

2. The system further includes a means for identifying a desired state, which is the state of the subject that the subject desires. The product recommendation device according to claim 1, wherein the determination means determines the recommended product based on the specified desired state and the estimated result of the health state.

3. The product recommendation device according to claim 2, wherein the determination means identifies nutrients necessary for the subject based on the identified desired state and the estimated result of the health state, and determines the recommended product containing the identified nutrients.

4. The estimation means further estimates the inner state of the subject based on the facial image, The product recommendation device according to claim 1, wherein the determination means determines the recommended product based on the estimated result of the health state and the estimated result of the internal state.

5. The acquisition means acquires, as relevant information related to the purchase of the subject, at least one of the subject's attribute information, the date and time information of the facial image, or purchase data showing the subject's past purchase history. The product recommendation device according to claim 1, wherein the determination means determines the recommended product based on the estimated health status and the related information.

6. The product recommendation device according to claim 1, further comprising a notification means for displaying or outputting audio information regarding the recommended product.

7. The product recommendation device according to claim 2, wherein the identifying means displays a plurality of candidates for the desired state together with the estimated result of the health state, and identifies the candidate selected from the plurality of candidates as the desired state.

8. The aforementioned determination means determines the recommended product based on a machine learning model that has learned the relationship between health status and products suitable for a person with that health status, and the estimated results of the health status. The product recommendation device according to claim 1, further comprising a learning means for acquiring purchase data showing the products purchased by the subject after the estimation of the health status, and for performing machine learning on the machine learning model based on the estimated health status and the purchase data.

9. Computers We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. Based on the estimated health status, a recommended product is determined for the subject. method.

10. We obtain facial images of the subjects, Based on the aforementioned facial image, the health status of the subject is estimated. A program that causes a computer to perform a process to determine recommended products to recommend to the subject based on the estimated results of the health status.

Images