Relationship estimation device and relationship estimation program

Abstract

To provide a relationship estimation device and a relationship estimation program capable of estimating the relationship of people appearing in photographed images with higher accuracy in comparison to conventional techniques.SOLUTION: A relationship estimation device 10 includes: an acquisition unit 11A that acquires photographed image information indicating a two-dimensional photographed image showing a plurality of visitors to a target facility and distance image information targeting a region including the plurality of visitors; an identification unit 11B that identifies, from the photographed image indicated by the photographed image information, a visitor group whose distances between the plurality of visitors obtained from the distance image information are equal to or less than a predetermined distance as a related visitor group having a relationship; and an estimation unit 11C that estimates the relationship of the plurality of visitors by inputting the image information corresponding to the image region of the related visitor group in the photographed image information to a relationship estimation model 13D pre-trained using the image information showing a photographed image of multiple people as input information and relationship information indicating the relationship of the plurality of people as output information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a relationship estimation device and a relationship estimation program.

Background Art

[0002] Conventionally, the following techniques have existed as techniques for estimating the relationship among visitors to a facility.

[0003] Patent Document 1 discloses a human relationship estimation device aimed at accurately estimating the human relationship between users.

[0004] This human relationship estimation device includes a detection unit that detects that two users are close to each other, a position attribute information acquisition unit that acquires information indicating different attributes for each of the users, associated with the position where proximity is detected by the detection unit, and a relationship estimation unit that estimates the human relationship between the users based on a combination of different attributes associated with each of the users, indicated by the information acquired by the position attribute information acquisition unit.

[0005] Patent Document 2 discloses a method for classifying the relationships of people from a set of photographs taken in a plurality of event groups.

[0006] This method includes a step of searching the set of photographs, identifying the people in the set of photographs, and determining the gender and age group of the identified people, and a step of estimating the social relationship between the identified people based on one set of rules using the occurrence and co-occurrence of the person, the gender of the person, and the age of the person.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] However, in the technique disclosed in Patent Document 1, there is a problem that it is difficult to accurately estimate the relationship between visitors when it is difficult to obtain position attribute information within the facility.

[0009] In addition, in the technique disclosed in Patent Document 2, there is a problem that it is difficult to accurately estimate the relationship between visitors when it is difficult to determine the gender or age group of the visitors.

[0010] The present disclosure has been made in view of the above facts, and an object thereof is to provide a relationship estimation device and a relationship estimation program that can estimate the relationship between people reflected in a captured image with higher accuracy compared to the conventional technique.

MEANS FOR SOLVING THE PROBLEMS

[0011] The relationship estimation device according to the present invention described in claim 1 includes an acquisition unit that acquires captured image information indicating a two-dimensional captured image in which a plurality of visitors who have visited the target facility are reflected, and distance image information indicating a distance image targeted at an area including the plurality of visitors, a specifying unit that specifies, as a related visitor group having a relationship, a visitor group in which the distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance in the captured image indicated by the captured image information, and an estimation unit that estimates the relationship between the plurality of visitors by inputting, to a relationship estimation model that has been preliminarily learned with image information indicating a captured image in which a plurality of people are reflected as input information and relationship information indicating the relationship between the plurality of people as output information, the image information corresponding to the image area of the related visitor group in the captured image information.

[0012] According to the relationship estimation device according to the present invention described in claim 1, imaging image information showing a two-dimensional imaging image in which a plurality of visitors who visited the target facility are reflected, and distance image information showing a distance image targeting an area including the plurality of visitors are acquired. From the imaging image shown by the imaging image information, a group of visitors whose distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance is specified as a related visitor group having a relationship. Image information showing an imaging image in which a plurality of people are reflected is used as input information, and image information corresponding to the image area of the related visitor group in the imaging image information is input to a relationship estimation model that has been previously learned with relationship information indicating the relationship of the plurality of people as output information, thereby estimating the relationship of the plurality of visitors. Compared with the prior art, the relationship of the people reflected in the imaging image can be estimated with higher accuracy.

[0013] The relationship estimation device according to the present invention described in claim 2 is the relationship estimation device according to claim 1, wherein the acquisition unit further acquires at least one of time information regarding the timing at which the estimation unit estimates the relationship and facility-related information related to the target facility, and the estimation unit corrects the relationship estimated by the relationship estimation model using at least one of the time information and the facility-related information.

[0014] According to the relationship estimation device according to the present invention described in claim 2, at least one of time information regarding the timing of estimating the relationship and facility-related information related to the target facility is further acquired, and by correcting the relationship estimated by the relationship estimation model using at least one of the time information and the facility-related information, the actual situation of the facility can be reflected in the estimation of the relationship.

[0015] The relationship estimation device according to the present invention described in claim 3 is the relationship estimation device according to claim 2, wherein the relationship estimation model further outputs probability information indicating the probability of the relationship indicated by the relationship information to be output as the output information, and the correction is a correction to the probability indicated by the probability information.

[0016] According to the relationship estimation device according to the present invention described in claim 3, the relationship estimation model further outputs, as output information, probability information indicating the probability of the relationship indicated by the relationship information to be output, and by making the above correction a correction for the probability indicated by the probability information, a correction that does not conflict with the definition of probability can be performed.

[0017] The relationship estimation device according to the present invention described in claim 4 is the relationship estimation device described in claim 2 or claim 3, wherein the time information is information indicating at least one of the day of the week and the time zone of the day when the estimation is performed, and the facility-related information is information indicating an event held at the target facility.

[0018] According to the relationship estimation device according to the present invention described in claim 4, by setting the time information as information indicating at least one of the day of the week and the time zone of the day when the estimation is performed, and setting the facility-related information as information indicating an event held at the target facility, the actual situation of the target facility can be reflected in the estimation of the relationship.

[0019] The relationship estimation device according to the present invention described in claim 5 is the relationship estimation device described in claim 1, further comprising a presentation unit that presents beneficial information predetermined to be beneficial for a plurality of people having the relationship estimated by the estimation unit to the plurality of visitors.

[0020] According to the relationship estimation device according to the present invention described in claim 5, by presenting beneficial information predetermined to be beneficial for a plurality of people having the estimated relationship to the plurality of visitors, the convenience for the visitors can be further improved.

[0021] The relationship estimation program according to the present invention described in claim 6 causes a computer to execute a process of acquiring shooting image information indicating a two-dimensional shooting image in which a plurality of visitors who visited the target facility are reflected, and distance image information indicating a distance image targeted at an area including the plurality of visitors, specifying, as a related visitor group having a relationship, a group of visitors whose distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance from the shooting image indicated by the shooting image information, inputting, to a relationship estimation model that has been pre-learned with image information indicating a shooting image in which a plurality of people are reflected as input information and relationship information indicating the relationship of the plurality of people as output information, image information corresponding to the image area of the related visitor group in the shooting image information, and estimating the relationship of the plurality of visitors.

[0022] According to the relationship estimation program according to the present invention described in claim 6, shooting image information indicating a two-dimensional shooting image in which a plurality of visitors who visited the target facility are reflected, and distance image information indicating a distance image targeted at an area including the plurality of visitors are acquired, a group of visitors whose distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance from the shooting image indicated by the shooting image information is specified as a related visitor group having a relationship, image information indicating a shooting image in which a plurality of people are reflected is used as input information, and by inputting, to a relationship estimation model that has been pre-learned with relationship information indicating the relationship of the plurality of people as output information, image information corresponding to the image area of the related visitor group in the shooting image information, and estimating the relationship of the plurality of visitors, it is possible to estimate the relationship of the people reflected in the shooting image with higher accuracy compared to the conventional technology.

Effects of the Invention

[0023] As described above, according to the present invention, it is possible to estimate the relationship of the people reflected in the shooting image with higher accuracy compared to the conventional technology.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0025] Hereinafter, an example of a relationship estimation system to which a relationship estimation device and a relationship estimation program according to the present invention are applied will be described in detail.

[0026] First, with reference to FIG. 1, the configuration of the relationship estimation system 90 according to the present embodiment will be described. FIG. 1 is a block diagram showing an example of the hardware configuration of the relationship estimation system 90 according to the present embodiment.

[0027] As shown in FIG. 1, the relationship estimation system 90 according to the present embodiment includes a relationship estimation device 10 and a digital signage device 60. Examples of the relationship estimation device 10 include information processing devices such as personal computers and server computers.

[0028] The relationship estimation system 90 according to this embodiment is a system for estimating the relationships of people passing through a predetermined passage of a commercial facility (hereinafter simply referred to as the "target facility") as an example of the target facility. While the relationship estimation device 10 is provided in the management room of the commercial facility, the digital signage device 60 is provided in the middle of the passage. Thus, in this embodiment, the case where a commercial facility is applied as the target facility will be described, but it is not limited to this form. For example, any facility such as public facilities like government offices, libraries, museums, community centers, town halls, etc., or educational facilities like universities, high schools, etc., that are used by unspecified people can be applied as the target facility.

[0029] The relationship estimation device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a memory 12 as a temporary storage area, a non-volatile storage unit 13, an input unit 14 such as a keyboard and a mouse, a display unit 15 such as a liquid crystal display, a media read / write device (R / W) 16, and a communication interface (I / F) unit 18. The CPU 11, the memory 12, the storage unit 13, the input unit 14, the display unit 15, the media read / write device 16, and the communication I / F unit 18 are connected to each other via a bus B. The media read / write device 16 reads information written on the recording medium 17 and writes information to the recording medium 17.

[0030] The storage unit 13 according to this embodiment is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, etc. A relationship estimation program 13A is stored in the storage unit 13 as a storage medium. The relationship estimation program 13A is stored (installed) in the storage unit 13 by setting the recording medium 17 on which the program 13A is written in the media read / write device 16 and the media read / write device 16 reading the program 13A from the recording medium 17. The CPU 11 appropriately reads the relationship estimation program 13A from the storage unit 13, expands it in the memory 12, and sequentially executes the processes included in the program 13A.

[0031] In addition, a recommended information database 13B and a facility information database 13C are stored in the storage unit 13. Details of the recommended information database 13B and the facility information database 13C will be described later.

[0032] In addition, a relationship estimation model 13D is stored in the storage unit 13 of the relationship estimation device 10 according to the present embodiment.

[0033] The relationship estimation model 13D according to the present embodiment is a model that has been pre-learned using, as input information, image information indicating a captured image in which a plurality of people are shown, and, as output information, relationship information indicating the relationships between the plurality of people. In particular, the relationship estimation model 13D according to the present embodiment is a machine learning model that has been pre-learned using, as input information, image information (hereinafter also referred to as "paired image information") indicating a pair of images of people created by cutting out the areas of people from the captured images shown by the captured image information of a plurality of people registered in a separately prepared learning database (not shown) using an object detection technique or the like, and, as output information, information (hereinafter referred to as "relationship information") indicating the relationship (parent-child, friends, colleagues, etc.) of the pair. Note that it is preferable to apply, as the captured image information registered in the learning database, the image information obtained by capturing with the camera 62 described later, but the present invention is not limited thereto.

[0034] The relationship estimation model 13D according to the present embodiment is a model based on AI (Artificial Intelligence) using a pre-trained CNN (Convolutional Neural Network), but the present invention is not limited thereto. For example, other machine learning models such as AI other than CNN, such as RNN (Recurrent Neural Network), may be applied as the relationship estimation model 13D.

[0035] Further, as shown in FIG. 1, a camera 62, a distance image sensor 64, and a display unit 66 such as a liquid crystal display provided in the digital signage device 60 are connected to the communication I / F unit 18 of the relationship estimation device 10 according to the present embodiment.

[0036] In the present embodiment, a camera 62 that captures a color moving image is applied, but the present invention is not limited to this. For example, as the camera 62, a form in which a camera that captures a monochrome moving image is applied may be used, or a form in which a camera that captures a color or monochrome still image is applied may be used. Further, in the present embodiment, a TOF (Time Of Flight) type sensor is applied as the distance image sensor 64, but the present invention is not limited to this.

[0037] FIG. 2 is a functional block diagram showing an example of the functional configuration of the relationship estimation system 90 according to the present embodiment.

[0038] As shown in FIG. 2, the relationship estimation device 10 according to the present embodiment includes an acquisition unit 11A, a specification unit 11B, an estimation unit 11C, and a presentation unit 11D. By the CPU 11 of the relationship estimation device 10 executing the relationship estimation program 13A, the CPU 11 functions as the acquisition unit 11A, the specification unit 11B, the estimation unit 11C, and the presentation unit 11D.

[0039] The acquisition unit 11A according to the present embodiment acquires captured image information indicating a two-dimensional captured image in which a plurality of visitors who have visited the target facility are reflected, and distance image information indicating a distance image targeted at an area including the plurality of visitors. The acquisition unit 11A according to the present embodiment acquires the captured image information from the camera 62 and acquires the distance image information from the distance image sensor 64.

[0040] Further, the specification unit 11B according to the present embodiment specifies, as a related visitor group having a relationship, a visitor group in which the distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance from the captured image indicated by the captured image information.

[0041] Note that in the relationship estimation system 90 according to the present embodiment, the horizontal distance between the top of the head of each visitor is applied as the distance between the plurality of visitors, but it is not limited thereto. For example, the horizontal distance between the center-of-gravity positions of the bodies of each visitor may be applied as the distance between the plurality of visitors. Further, in the relationship estimation system 90 according to the present embodiment, as the predetermined distance, the distance obtained in advance by experiments or the like (50 cm in the present embodiment) is fixedly applied on the assumption that there is a high possibility that people existing at a distance equal to or less than the distance have some relationship, but it is not limited thereto. For example, the administrator or the like of the relationship estimation system 90 may appropriately input according to the use of the relationship estimation system 90, the required relationship estimation accuracy, and the like.

[0042] Further, the estimation unit 11C according to the present embodiment estimates the relationship of the plurality of visitors by inputting the image information corresponding to the image area of the relevant visitor group in the captured image information to the relationship estimation model 13D. In the relationship estimation system 90 according to the present embodiment, the captured image information indicating a still image of a single frame is extracted from the captured image information acquired in real time from the camera 62, and the pair image information regarding the person included in the captured image indicated by the captured image information and included in the common relevant visitor group is applied as the input information.

[0043] Then, the presentation unit 11D according to the present embodiment presents beneficial information (corresponding to the recommended information described later in the present embodiment) predetermined as being beneficial for the plurality of people having the relationship estimated by the estimation unit 11C to the plurality of visitors. Note that in the present embodiment, as the presentation by the presentation unit 11D, the presentation by the display by the display unit 66 is applied, but it is not limited thereto. For example, the presentation by the voice by the voice playback device, the presentation by the printing by the image forming device, or the like may be applied as the presentation by the presentation unit 11D.

[0044] In addition, the acquisition unit 11A according to the present embodiment further acquires at least one of the timing information regarding the timing when the estimation unit 11C performs the relationship estimation and the facility-related information related to the target facility (in the present embodiment, both, corresponding to the facility information described later), and the estimation unit 11C corrects the relationship estimated by the relationship estimation model 13D using at least one of the timing information and the facility-related information (in the present embodiment, both).

[0045] In particular, the relationship estimation model 13D according to the present embodiment is further configured to output, as output information, probability information indicating the probability of the relationship indicated by the output relationship information, and as the above correction, apply a correction to the probability indicated by the probability information.

[0046] Note that in the relationship estimation system 90 according to the present embodiment, as the timing information, information indicating at least one of the day of the week and the time zone of the day of estimation (in the present embodiment, only the time zone) is applied, and as the facility-related information, information indicating an event held at the target facility is applied, but neither the timing information nor the facility-related information is limited to these.

[0047] Next, with reference to FIG. 3, the recommended information database 13B according to the present embodiment will be described. FIG. 3 is a schematic diagram showing an example of the configuration of the recommended information database 13B according to the present embodiment. The recommended information database 13B stores information recommended to visitors to the target facility by the digital signage device 60 in the relationship estimation system 90 according to the present embodiment.

[0048] As shown in FIG. 3, the recommended information database 13B according to the present embodiment stores each piece of information on the relationship and the recommended information in an associated manner.

[0049] The above relationship is information indicating the relationship of the above-described pair, and the above recommendation information is information indicating information recommended for a pair having a corresponding relationship with respect to the target facility. In the present embodiment, as the recommendation information, information indicating the name of a store recommended for a pair having a corresponding relationship is applied for each use type such as a restaurant, a clothing store, etc. provided in a commercial facility as the target facility, and is shown in the order of candidates. In the example shown in FIG. 3, for each type of relationship, information up to the second candidate is registered for each recommended target (in the example shown in FIG. 3, restaurants, clothing stores, etc.), but it is not limited to this.

[0050] Next, with reference to FIG. 4, the facility information database 13C according to the present embodiment will be described. FIG. 4 is a schematic diagram showing an example of the configuration of the facility information database 13C according to the present embodiment. The facility information database 13C stores information regarding events implemented in the target facility.

[0051] As shown in FIG. 4, in the facility information database 13C according to the present embodiment, information on each of the event name and schedule is stored in an associated manner.

[0052] The above event name is information indicating the name of an event implemented in the target facility, and the above schedule is information indicating the period and time zone during which the event corresponding to the event name is implemented. Note that the information indicating the event name corresponds to the above-described facility-related information, and the information indicating the schedule corresponds to the above-described time information. Thus, in the facility information database 13C according to the present embodiment, for each of the above events, the period and time zone for implementation are registered, but it is not limited to this. For example, for each of the above events, the day of the week for implementation may be registered.

[0053] Next, with reference to FIGS. 5 to 6, a process of estimating a relationship using the relationship estimation model 13D according to the present embodiment (hereinafter referred to as "relationship estimation process") will be described. FIG. 5 is a schematic diagram for explaining the relationship estimation process by a conventional machine learning model, and FIG. 6 is a schematic diagram for explaining the relationship estimation process by the relationship estimation model 13D according to the present embodiment.

[0054] As shown in FIG. 5, in the learning phase of the machine learning model used in the conventional relationship estimation process, from the captured image information shown in the captured image in which a plurality of people are shown, the area of the person is cut out using a conventionally known object detection technique or the like to create pair image information indicating a pair of images of the person. And in this learning phase, the created pair image information is used as input information, and the relationship information indicating the relationship of the pair shown by the pair image is machine-learned in advance as output information.

[0055] On the other hand, in the operation phase of the machine learning model in this conventional relationship estimation process, relationship information is estimated together with its probability by inputting the pair image information created by cutting out the area of the person from the captured image information obtained by the camera in the same manner as in the learning phase into the machine learning model.

[0056] On the other hand, the learning phase of the relationship estimation model 13D used in the relationship estimation process according to the present embodiment is performed in the same manner as the above-described conventional technique. In contrast, in the operation phase of the relationship estimation model 13D according to the present embodiment, the area of the person is cut out from the captured image (described as "RGB image" in FIG. 6) shown by the captured image information obtained by the camera 62 by a conventionally known object detection technique or the like. And in this operation phase, the pair image information indicating the pair of people included in the common related visitor group obtained from the distance image information in the cut-out image of the person is applied as input information.

[0057] As a result, the input information can be made into pair image information of a combination of people highly likely to have a relationship, and thus the estimation accuracy of the relationship by the relationship estimation model 13D can be significantly improved.

[0058] Also, the relationship estimation model 13D according to the present embodiment is also configured to output, as output information, probability information indicating the probability of the relationship indicated by the output relationship information, similar to the conventional machine learning model described above. In the relationship estimation process according to the present embodiment, the probability indicated by the probability information is weighted using the time information and the facility-related information (described as "external information" in FIG. 6), thereby correcting the probability. In this way, in the relationship estimation process according to the present embodiment, since the probability estimated by the relationship estimation model 13D is corrected using external information as post-processing, it is possible to reflect the actual situation of the target facility in the relationship estimation after performing a correction that does not conflict with the definition of the probability.

[0059] In the present embodiment, as the above correction, when the tendency of visitors is known in advance from the time zone of the day or the event held at the target facility, correction probabilities and weights for each relationship are set, and the classification probability of the relationship is weighted. When there is no prior knowledge, the classification probability of the relationship is not corrected by setting the weight to 0 (zero).

[0060] Next, with reference to FIGS. 7 to 8, the operation of the relationship estimation device 10 according to the present embodiment will be described. FIG. 7 is a flowchart showing an example of the flow of the relationship estimation process according to the present embodiment. The relationship estimation process starts when the business of the target facility starts. Here, for the sake of avoiding complication, the case where the relationship estimation model 13D has been machine-learned and the recommendation information database 13B and the facility information database 13C have been constructed will be described.

[0061] In step 100 shown in FIG. 7, the CPU 11 reads all the information (hereinafter referred to as "recommendation information") from the recommendation information database 13B and reads all the information (hereinafter referred to as "facility information") from the facility information database 13C.

[0062] In step 102, the CPU 11 acquires shooting image information for one frame from the camera 62 and distance image information for one frame from the distance image sensor 64.

[0063] In step 104, as described above, the CPU 11 extracts all pair image information taking into account relevant visitor groups from the acquired shooting image information.

[0064] In step 106, the CPU 11 inputs all the extracted pair image information into the relationship estimation model 13D as described above, and estimates the relationships corresponding to all the pair image information. At this time, the CPU 11 performs the above-described correction on the estimated relationships using the read facility information.

[0065] In step 108, the CPU 11 specifies recommended information corresponding to the relationships corresponding to all the pair image information obtained by the above processing, and controls the display unit 66 of the digital signage device 60 to display a recommended screen having a predetermined configuration using the specified recommended information.

[0066] FIG. 8 shows an example of the recommended screen according to the present embodiment. As shown in FIG. 8, in the recommended screen according to the present embodiment, information 15A indicating the estimated relationship and information 15B that is desired to be recommended to people having the corresponding relationship are displayed. At this time, if this time zone corresponds to the lunch and dinner time zones, the CPU 11 displays recommended information regarding restaurants, and if it is other time zones, displays recommended information suitable for that time zone, such as clothing stores.

[0067] As a result, meaningful information can be obtained for visitors, so that customer satisfaction can be improved, and ultimately it is also a great merit for the facility operator.

[0068] In step 110, the CPU 11 determines whether the end timing, which is predetermined as the timing to end the relationship estimation process, has arrived. If the determination is negative, it returns to step 102, while if the determination is positive, it ends this relationship estimation process. Note that in this embodiment, as the above end timing, the timing when the business of the target facility ends is applied, but it is not limited to this.

[0069] As described above, according to this embodiment, imaging image information indicating a two-dimensional captured image in which a plurality of visitors who visited the target facility are reflected, and distance image information indicating a distance image targeted at an area including the plurality of visitors are acquired. From the captured image indicated by the imaging image information, a group of visitors whose distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance is specified as a related visitor group having a relationship. Image information indicating a captured image in which a plurality of people are reflected is used as input information, and image information corresponding to the image area of the related visitor group in the imaging image information is input to a relationship estimation model that has been pre-learned with the relationship information indicating the relationship of the plurality of people as output information, thereby estimating the relationship of the plurality of visitors. Therefore, compared with the conventional technology, the relationship of the people reflected in the captured image can be estimated with higher accuracy.

[0070] Thus, according to this embodiment, compared with the conventional technology, the estimation accuracy of the relationship can be further improved, so the following secondary effects can be expected.

[0071] First, it leads to the development of marketing services that utilize the estimated relationship. Second, it leads to the discovery of potential needs that visitors in the target facility are not aware of. Third, in the analysis of the flow of people, which has attracted attention in the marketing of construction and real estate businesses, the relationship between multiple people can also be applied as important data.

[0072] Further, according to the present embodiment, at least one of time information regarding the timing of performing the relationship estimation and facility-related information related to the target facility is further acquired, and correction is performed on the relationship estimated by the relationship estimation model using at least one of the time information and the facility-related information. Therefore, the actual situation of the facility can be reflected in the relationship estimation.

[0073] Further, according to the present embodiment, the relationship estimation model is configured to further output, as output information, probability information indicating the probability of the relationship indicated by the relationship information to be output, and the above correction is performed as a correction to the probability indicated by the probability information. Therefore, a correction that does not conflict with the definition of probability can be performed.

[0074] Further, according to the present embodiment, the time information is information indicating at least one of the day of the week and the time zone of the day when the above estimation is performed, and the facility-related information is information indicating an event held at the target facility. Therefore, the actual situation of the target facility can be reflected in the relationship estimation.

[0075] Furthermore, according to the present embodiment, beneficial information predetermined to be beneficial for a plurality of people having the estimated relationship is presented to a plurality of visitors. Therefore, the convenience for the visitors can be further improved.

[0076] In the above embodiment, the case where pair image information is applied as input information to the relationship estimation model 13D has been described, but the present invention is not limited to this. For example, a form in which image information including three or more people is applied as input information to the relationship estimation model 13D may also be used.

[0077] Also, in the above embodiment, the case where the camera 62, the distance image sensor 64, and the display unit 66 provided in the digital signage device 60 are applied has been described, but the present invention is not limited to this. For example, a form in which the camera 62, the distance image sensor 64, and the display unit 66 are each prepared individually and applied may also be used.

[0078] Also, in the above-described embodiment, for example, as the hardware structure of the processing unit that executes the respective processes of the acquisition unit 11A, the specification unit 11B, the estimation unit 11C, and the presentation unit 11D, various types of processors shown below can be used. As described above, among the various types of processors, in addition to the CPU which is a general-purpose processor that executes software (program) and functions as a processing unit, there are also programmable logic devices (PLDs) such as FPGAs (Field-Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electric circuits, etc., which are processors having a circuit configuration specifically designed to execute specific processes such as ASICs (Application Specific Integrated Circuits).

[0079] The processing unit may be configured by one of these various types of processors, or may be configured by a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the processing unit may be configured by one processor.

[0080] As an example of configuring the processing unit by one processor, firstly, as represented by computers such as clients and servers, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor functions as the processing unit. Secondly, as represented by System On Chip (SoC), etc., there is a form in which a processor that realizes the functions of the entire system including the processing unit with one IC (Integrated Circuit) chip is used. Thus, the processing unit is configured using one or more of the above various types of processors as its hardware structure.

[0081] Furthermore, as the hardware structure of these various types of processors, more specifically, an electric circuit (circuitry) combining circuit elements such as semiconductor elements can be used.

Explanation of Symbols

[0082] 10 Relationship Estimation Device 11 CPU 11A Acquisition Unit 11B Identification Unit 11C Estimation Unit 11D Presentation Unit 12 Memory 13 Storage Unit 13A Relationship Estimation Program 13B Recommended Information Database 13C Facility Information Database 13D Relationship Estimation Model 14 Input Unit 15 Display Unit 15A Information Indicating Relationship 15B Information to be Recommended 16 Medium Read / Write Device 17 Recording Medium 18 Communication I / F Unit 60 Digital Signage Device 62 Camera 64 Distance Image Sensor 66 Display Unit 90 Relationship Estimation System

Claims

1. An acquisition unit that acquires imaging image information indicating a two-dimensional captured image in which a plurality of visitors who have visited the target facility are shown, and distance image information indicating a distance image targeted at an area including the plurality of visitors; A specifying unit that specifies, as a related visitor group having a relationship, a visitor group in which the distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance from the captured image indicated by the captured image information; An estimation unit that inputs, to a relationship estimation model that has been pre-learned with image information indicating a captured image in which a plurality of people are shown as input information and relationship information indicating the relationship of the plurality of people as output information, the image information corresponding to the image area of the related visitor group in the captured image information, thereby estimating the relationship of the plurality of visitors; A relationship estimation device comprising the above.

2. The acquisition unit further acquires at least one of time information regarding the timing at which the estimation unit estimates the relationship and facility-related information related to the target facility, The estimation unit corrects the relationship estimated by the relationship estimation model using at least one of the time information and the facility-related information. The relationship estimation device according to Claim 1.

3. The relationship estimation model further outputs, as the output information, probability information indicating the probability of the relationship indicated by the relationship information to be output, The correction is a correction to the probability indicated by the probability information. The relationship estimation device according to Claim 2.

4. The time information is information indicating at least one of the day of the week and the time zone on the day when the estimation is performed, The facility-related information is information indicating an event held at the target facility. The relationship estimation device according to Claim 2 or Claim 3.

5. The relationship estimation device according to Claim 1, further comprising a presentation unit that presents beneficial information determined in advance as being beneficial to a plurality of people having the relationship estimated by the estimation unit for the plurality of visitors. The relationship estimation device according to Claim 1.

6. Acquire imaging image information indicating a two-dimensional captured image in which a plurality of visitors who have visited the target facility are shown, and distance image information indicating a distance image targeted at an area including the plurality of visitors, Specify, as a related visitor group having a relationship, a visitor group in which the distance between the plurality of visitors obtained from the distance image information is equal to or less than a predetermined distance from the captured image indicated by the captured image information. Using, as input information, image information indicating a photographed image in which a plurality of people are shown, and inputting, to a relationship estimation model that has been pre-learned with relationship information indicating the relationships of the plurality of people as output information, the image information corresponding to the image area of the relevant visitor group in the photographed image information, thereby estimating the relationships of the plurality of visitors. A relationship estimation program for causing a computer to execute the process.

Images