Information processing device, information processing program, information processing method, and information processing system

Abstract

This makes it possible to provide information to target individuals that is tailored to the organizations or people they are interested in or concerned about. [Solution] The system includes an image acquisition unit (101) that acquires captured images of a subject; an object recognition unit (102) that recognizes the subject's possessions or clothing based on the captured images and detects trademarks or designs contained in the recognized possessions or clothing; an orientation estimation unit (103) that estimates the group or person of interest or concern of the subject based on object recognition information indicating the detected trademark or design; a content selection unit (104) that selects content to be provided from among content related to the group or person based on the orientation information indicating the group or person; a content acquisition unit (105) that acquires the selected content; and a display control unit (106) that displays the acquired content on a display unit (201).

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing program, an information processing method, and an information processing system.

Background Art

[0002] Conventionally, there has been known a technique for providing information to a person (hereinafter referred to as a "target person") near a display device by utilizing a display device such as a sign. For example, Patent Document 1 discloses an information presentation system that, in a vehicle, based on captured video images captured by an in-vehicle camera, estimates the customer's interesting items from the customer's belongings (e.g., luggage) at the time of boarding, and distributes information such as information on recommended stores to an in-vehicle terminal or the customer's mobile terminal. For example, when the information presentation system analyzes that a passenger has a telephoto lens or a single-lens reflex camera as a belonging, it estimates that the passenger is interested in cameras, and distributes information on camera stores or large home appliance stores as recommended stores to the in-vehicle terminal or the customer's mobile terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In information provision using a display device such as a sign, effective operation is required. One of the information provisions for which effective operation is required is provision of information on a group or a person in which the target person has an interest or concern, to the target person. Such information provision is likely to attract the interest or concern of the target person, and also leads to acquisition of supporters for the group or the person. In order for the information provided to be as effective as described above, it is important that the information provided to the target audience is about organizations or individuals that the target audience is interested in or concerned with. For this to be possible, it is necessary to be able to estimate which organizations or individuals the target audience is interested in or concerned with. The conventional technologies described above do not estimate the organizations or individuals that the target user is interested in. As a result, conventional technologies have the problem of not being able to provide information tailored to the organizations or individuals that the target user is interested in.

[0005] This disclosure was made to solve the above-mentioned problems, and aims to provide an information processing device that enables information provision using a display device to be provided to a target person in a manner that is tailored to the organizations or individuals that the target person is interested in or concerned with. [Means for solving the problem]

[0006] The information processing device of this disclosure includes: an image acquisition unit that acquires an image of a subject; an object recognition unit that recognizes the subject's possessions or clothing based on the image acquired by the image acquisition unit and detects existing trademarks or designs contained in the recognized possessions or clothing; an orientation estimation unit that estimates an organization or person of interest or concern to the subject based on object recognition information indicating the trademark or design detected by the object recognition unit; a content selection unit that selects content to be provided from content related to the organization or person stored in a content server based on orientation information indicating the organization or person estimated by the orientation estimation unit; a content acquisition unit that acquires the content selected by the content selection unit from the content server; and a display control unit that causes the content acquired by the content acquisition unit to be displayed on a display unit. [Effects of the Invention]

[0007] According to this disclosure, the information processing device is configured as described above, so that when providing information using the display device, it is possible to provide information to the target person that is tailored to the organizations or individuals that the target person is interested in or concerned about. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the information processing device according to Embodiment 1. [Figure 2] This is a diagram illustrating the general appearance of the information output device according to Embodiment 1. [Figure 3] This is a flowchart illustrating the operation of the information processing device according to Embodiment 1. [Figure 4] This figure shows an example configuration of an information processing device that includes an attribute determination unit in Embodiment 1. [Figure 5] This is a flowchart illustrating the operation of an information processing device equipped with an attribute determination unit in Embodiment 1. [Figure 6] This figure shows an example configuration of an information processing device that includes a questioning unit in Embodiment 1. [Figure 7] This is a flowchart illustrating the operation of an information processing device equipped with a questioning unit in Embodiment 1. [Figure 8] This figure shows an example configuration of an information output device equipped with a microphone and a speaker in Embodiment 1. [Figure 9] This diagram illustrates the general appearance of an information output device equipped with a microphone and a speaker in Embodiment 1. [Figure 10] This figure shows an example configuration of an information output device equipped with a character section in Embodiment 1. [Figure 11] This figure illustrates the general appearance of an information output device equipped with a character section in Embodiment 1. [Figure 12]This is a diagram showing a configuration example of an information output device provided with a moving mechanism in Embodiment 1. [Figure 13] This is a diagram for explaining an overview of the appearance of an information output device provided with a moving mechanism in Embodiment 1. [Figure 14] This is a diagram showing a configuration example of an information processing device provided with a position optimization unit in Embodiment 1. [Figure 15] This is a flowchart for explaining the operation of an information processing device provided with a position optimization unit in Embodiment 1. [Figure 16] FIG. 16A and FIG. 16B are diagrams showing an example of the hardware configuration of an information processing device according to Embodiment 1.

Embodiments for Carrying out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram showing a configuration example of an information processing device 1 according to Embodiment 1. The information processing device 1 is mounted on, for example, a server (not shown). The information processing device 1 is communicably connected to an information output device 2 via a network. The information processing device 1 and the information output device 2 constitute an information processing system 10. In addition, the information processing device 1 is communicably connected to an activity subject DB (database) 3, a content server 4, and a fan profile DB 5 via a network.

[0010] The activity entity DB stores information related to groups or individuals that conduct various activities such as business activities or social activities (hereinafter referred to as "activity entity information"). In Embodiment 1, the groups or individuals that conduct various activities such as business activities or social activities are, for example, sports teams, schools, companies, artists who are active in groups, players belonging to sports teams, members of artists who are active in groups, artists who are active as individuals, etc. The activity entity information includes, for example, information in which a group or individual that conducts various activities such as business activities or social activities is associated with information regarding a trademark or design related to the group or individual. Details of the activity entity information will be described later with an example. In Embodiment 1, the "trademark" refers to among those that can be recognized by human perception, letters, figures, symbols, three-dimensional shapes, colors, or combinations thereof, sounds, etc. The "design" refers to the shape, pattern, color, or combination thereof of an article, building, or image that causes aesthetic feeling through vision. Note that the "article" includes parts of the article, and the "building" includes parts of the building. Taking specific examples, in Embodiment 1, the trademark or design includes, for example, letters, numbers, colors, patterns, logo marks, characters, designs, etc. The trademark or design refers to these visual elements.

[0011] In the following Embodiment 1, the group or individual that conducts various activities such as business activities or social activities is also simply referred to as "group or individual".

[0012] The content server 4 stores various contents. Examples of the contents stored in the content server 4 include images or videos related to a group or individual, advertising images or videos of sponsors of the group or individual, promotional images or videos of sponsors of the group or individual, guidance images or videos from local governments or public institutions, etc. For example, a plurality of consecutive videos with different contents may be stored in the content server 4 as one content. The storage of various types of content on content server 4 is performed, for example, by an administrator. The administrator prepares content that could potentially be provided to specific target individuals in advance and stores it on content server 4. The definition of "specific target individuals" will be described later. Furthermore, the content stored in content server 4 is assigned information that allows it to be identified (for example, an ID). Here, we will assume that the content is assigned an ID, and the ID assigned to the content will also be referred to as the "content ID".

[0013] Fan Profile DB5 manages information that indicates individual characteristics (hereinafter referred to as "profile information"). Details of profile information will be explained later with an example. Note that in Figure 1, the information processing device 1 is shown to be connected to the fan profile DB5, but it is not mandatory for the information processing device 1 to be connected to the fan profile DB5.

[0014] The information processing device 1 acquires an image of the subject captured by the imaging unit 202 of the information output device 2, and estimates the group or person that is the subject's interest or concern based on the acquired image. Then, based on the information indicating the estimated group or person (hereinafter referred to as "interest information"), the information processing device 1 selects content to provide from the content related to the group or person stored in the content server 4, retrieves the selected content from the content server 4, and displays it on the display unit 201 of the information output device 2. Here, "subject" refers to a person present around the information output device 2, or more specifically, the display unit 201 on which the content is displayed, and it is not necessary for the "subject" to be identified as an individual. Furthermore, when the information processing device 1 displays content on the display unit 201, it can improve the accuracy of estimating the group or person that is the subject of the subject's interest or concern, based on the subject's reaction to the displayed content. Furthermore, when the information processing device 1 displays content on the display unit 201, it outputs information related to the displayed content, or information indicating the subject's reaction to the content, as profile information to the fan profile DB 5. A detailed configuration example of the information processing device 1 will be described later.

[0015] Here, the general appearance of the information output device 2 in Embodiment 1 will be described with reference to the drawings. Figure 2 is a diagram illustrating the general appearance of the information output device 2 according to Embodiment 1. Figure 2 shows the information output device 2 viewed from the front. As shown in Figure 2, the information output device 2 comprises a main unit (indicated as "M" in Figure 2; not shown in Figure 1), a display unit 201 provided on the main unit, and an imaging unit 202 provided on the main unit. The display unit 201 is a video display device such as a digital signage display. In the following Embodiment 1, the display unit 201 is assumed to be a digital signage display, but this is merely an example, and the display unit 201 may be a liquid crystal display or the like. The imaging unit 202 is an imaging device that captures images of the area surrounding the display unit 201, including at least the front of the display unit 201.

[0016] In Figure 2, the main unit is shown as a housing, but the shape of the main unit is not specified. It is sufficient that the information output device 2 includes a display unit 201 and an imaging unit 202. Furthermore, in Figure 2, the imaging unit 202 is shown to be located above the display unit 201 when the display unit 201 is viewed from the front with the information output device 2 installed, but this is merely one example. For example, the imaging unit 202 may be located to the right or left of the display unit 201 when the display unit 201 is viewed from the front with the information output device 2 installed. The imaging unit 202 only needs to be configured to capture an area that includes at least the front of the display unit 201.

[0017] In Figure 1, the information processing system 10 is shown to have only one information output device 2, but this is merely an example. In the information processing system 10, multiple information output devices 2 may be installed in appropriate locations, and multiple information output devices 2 may be connected to the information processing device 1 via a network. The information processing device 1 can control the display of different content on the display unit 201 for multiple information output devices 2.

[0018] An example of the configuration of the information processing device 1 according to Embodiment 1 will be described. As shown in Figure 1, the information processing device 1 comprises an image acquisition unit 101, an object recognition unit 102, an orientation estimation unit 103, a content selection unit 104, a content acquisition unit 105, a display control unit 106, a reaction information acquisition unit 107, an effect estimation unit 108, a model update unit 109, a model storage unit 110, and a profile update unit 111.

[0019] The image acquisition unit 101 acquires captured images of the subject from the imaging unit 202. The image acquisition unit 101 outputs the acquired image to the object recognition unit 102.

[0020] The object recognition unit 102 recognizes the subject's possessions or clothing based on the captured image acquired by the image acquisition unit 101, and detects any marks or designs included in the recognized possessions or clothing. In Embodiment 1, "included in possessions or clothing" means existing as part of the possessions or clothing. Existing as part of possessions or clothing includes the addition of visual information. Methods for adding visual information include various methods such as printing, embroidery, sewing, and heat transfer. That is, the pattern on possessions or clothing can be said to be included in possessions or clothing, and for example, the number on the back of a uniform can be said to be included in the uniform, which is clothing, if the number is visible on the uniform through various methods such as printing, embroidery, or sewing on a patch.

[0021] The object recognition unit 102 may, for example, use known image recognition techniques to recognize the subject's possessions or clothing in the captured image and detect any marks or designs contained therein. The object recognition unit 102 outputs information indicating the detected trademark or design (hereinafter referred to as "object recognition information") to the orientation estimation unit 103.

[0022] Object recognition information is, for example, the captured image of the portion in which the detected trademark or design is captured. For example, the object recognition unit 102 may set a minimum rectangular area surrounding the detected trademark or design on the captured image and use the captured image of the portion of the set area as object recognition information, or it may use information relating the captured image to the coordinates of the minimum rectangle on the captured image as object recognition information.

[0023] Furthermore, the object recognition information may be, for example, the captured image of the portion of the possession or wearable item that contains the detected trademark or design. For example, the object recognition unit 102 may set a minimum rectangular area on the captured image that surrounds the possession or wearable item that contains the detected trademark or design, and use the captured image of the portion of the set area as object recognition information, or it may use information that associates the captured image with the coordinates of the minimum rectangle on the captured image as object recognition information.

[0024] Furthermore, the object recognition information may also include, for example, information indicating the outline or geometric features (circle, square, etc.) of the detected trademark or design, information indicating the main color or color distribution pattern of the detected trademark or design, or information indicating the string of characters or the type of font of the characters included in the trademark or design.

[0025] The object identification information only needs to be information that clearly identifies the characteristics of the trademark or design contained in the recognized possession or garment. Object recognition information may include, for example, information indicating the subject. This information may include, for example, information indicating the subject's position on the captured image. The subject's position may be represented, for example, by the coordinates of the center of the smallest rectangular area surrounding the subject on the captured image. Alternatively, the information indicating the subject may be, for example, an image from which the subject's face region has been extracted. It should be noted that the information indicating the subject only needs to indicate which person it is on the captured image, and does not need to be information that identifies the individual subject.

[0026] The orientation estimation unit 103 estimates the group or person that is the subject of the subject's interest or concern, based on the object recognition information output from the object recognition unit 102. For example, the orientation estimation unit 103 compares the activity entity information stored in the activity entity DB3 with the object recognition information to estimate the group or person that is the subject of the subject's interest or concern.

[0027] Information about the entity involved in the activity includes, for example, information such as (1) to (6) below. Note that the following is merely an example, and the information about the entity involved only needs to include information that associates a trademark or design with information that can identify an organization or person. (1) Sports team information that associates information that can identify a sports team (e.g., ID, team name, etc.) with information that shows the logo or uniform design of the sports team. (2) Player information that associates information that can identify a player (e.g., ID, player name, etc.) with information that indicates the name of the team to which the player belongs, the pattern of the team's uniform, the player's name, or the player's number. (3) Artist information that associates information that can identify the artist (e.g., ID, artist name, etc.) with information that shows the artist's logo, group name, artist name, members, etc. (4) Facial information, including images of the faces of athletes or artists. (5) School information that associates information that can identify a school (e.g., ID, school name, etc.) with information that shows the design of the school emblem, logo, uniform, or gym clothes, etc. (6) Event information including information indicating the date and time of the match or other event such as a live event.

[0028] For example, regarding (1), suppose that information that can identify sports team A and sports team information, which is an image of sports team A's logo or uniform, are stored in the activity entity DB3 as activity entity information. For example, if the object recognition information is an image of a uniform with sports team A's logo, the orientation estimation unit 103 matches the activity entity information with the object recognition information and identifies sports team A from the logo shown in the image in the object recognition information. The orientation estimation unit 103 then estimates that the identified sports team A is the object of the subject's interest or concern. Generally, when a person owns or wears an item that includes a certain mark or design, that person is presumed to have an interest in or concern for the entity or person identified by that mark or design.

[0029] Furthermore, for example, regarding (2), suppose that player information, which includes information that can identify player X, information indicating player X's jersey number "10", and an image of the uniform of sports team B to which player X belongs, is stored in the activity entity DB3 as activity entity information. For example, if the object recognition information is an image of sports team B's uniform with the number "10" printed on it, the orientation estimation unit 103 matches the activity entity information with the object recognition information and identifies player X from the number "10" of sports team B shown in the image in the object recognition information. The orientation estimation unit 103 then estimates that the identified player X is the object of the subject's interest or concern.

[0030] For example, a comparison rule is defined in advance by an administrator or other person to determine, in what order, and how to compare the content of object recognition information with the activity entity information. Information indicating this comparison rule (hereinafter referred to as "comparison rule information") is stored in a memory unit not shown in the diagram. The orientation estimation unit 103 refers to the comparison rule information and, according to the comparison rule, estimates the object of the subject's interest or concern from the activity entity information and object recognition information.

[0031] The method described above is merely one example, and the orientation estimation unit 103 may estimate the subject's interests or concerns using other methods. For example, the orientation estimation unit 103 may use a pre-trained machine learning model (hereinafter referred to as the "machine learning model") to estimate the object of the subject's interest or concern. The machine learning model here is assumed to be a model trained through supervised learning, which takes object recognition information as input and outputs information indicating the object of interest or concern. The information indicating the object of interest or concern output by the machine learning model is an ID or image, or other information that identifies the organization or object of interest or concern. If the orientation estimation unit 103 does not use activity subject information to estimate the subject's interests or concerns, the information processing device 1 is not required to be connected to the activity subject DB3.

[0032] The orientation estimation unit 103 outputs information indicating the group or person that it estimates to be of interest or concern to the subject (hereinafter referred to as "orientation information") to the content selection unit 104. Personality information, for example, is information that can identify a group or person that is presumed to be of interest or concern to the subject. In the orientation information, information indicating the subject is associated with information indicating a group or person presumed to be the subject's interest or concern. As mentioned above, the information indicating the subject is, for example, information indicating the subject's position in the captured image. The information indicating the subject may also be from an captured image in which the subject's face region has been extracted. The information indicating the subject does not have to be information that identifies the individual subject. Furthermore, information indicating organizations or individuals that are presumed to be of interest to the subject may include, for example, identifiable information about organizations or individuals included in the activity entity information, or information indicating the object of interest output by a machine learning model.

[0033] The content selection unit 104 selects content to be provided from among the content related to the group or person stored in the content server 4, based on the orientation information indicating the group or person estimated by the orientation estimation unit 103. In Embodiment 1, the content selection unit 104 is assumed to select the content to be provided using a machine learning model that employs reinforcement learning (hereinafter referred to as the "reinforcement learning model"). In Embodiment 1, when the content selection unit 104 "selects content," it does not mean selecting actual content, but rather selecting information that can identify the content (for example, an ID). The content selection unit 104 selects content by providing orientation information to the reinforcement learning model and obtaining information that allows for content identification. The reinforcement learning model selects the content that is expected to yield the highest reward from among the content stored in the content server 4 and outputs the content ID assigned to the selected content. The content selection unit 104 obtains the content ID output by the reinforcement learning model. The reinforcement learning model is stored in the model memory unit 110.

[0034] Here, the content selection unit 104, based on the orientation estimation result of the orientation estimation unit 103, determines that the target person is a target person who is eligible to receive content tailored to their interests (hereinafter referred to as a "specific target person"), and selects content related to the group or person estimated by the orientation estimation unit 103 based on the orientation information. If the content selection unit 104 determines that the target person is not a specific target person, it selects the content that is normally set as content.

[0035] The content selection unit 104 determines, for example, whether the subject is a specific subject based on the orientation information, which is the estimation result of a group or person by the orientation estimation unit 103, according to the conditions for determining a specific subject. The criteria for determining a person as a target individual are defined by the conditions under which a person is deemed to be a target individual if the organization or person is the object of their interest or concern. These criteria for determining a person as a target individual are set in advance by the administrator or other relevant personnel. For example, an administrator may decide that an organization or individual with whom the administrator has a contract, or an organization or individual related to a company with which the administrator has a contract, is the target for providing content to appeal to people (target audiences) who are interested in that organization or individual. The organization or individual that is the target for providing content to appeal to people (target audiences) who are interested in that organization or individual is also called the "target activity entity." The administrator then sets a condition for determining the target audience that the designated organization or individual is presumed to be interested in is designated as the "specific target audience." To give a concrete example, suppose the administrator currently has a contract with sports team A. In this case, the administrator decides that sports team A is the target to which content will be provided to appeal to the target audience. The administrator then sets the condition that "the target audience is those who are presumed to be of interest or concern to sports team A" as a condition for determining the target audience.

[0036] When administrators set conditions for identifying specific individuals, they store information indicating these conditions (hereinafter referred to as "conditions for identifying specific individuals information") in a memory unit not shown in the diagram. Furthermore, administrators may add or modify the criteria information for identifying specific individuals as appropriate. For example, if administrators enter into a new contract with an organization, person, or company, they may add a condition to the criteria for identifying specific individuals that designates individuals who are presumed to be of interest or concern to the newly contracted organization or person, or to an organization or person related to the newly contracted company, as specific individuals. Furthermore, for example, an administrator may set the above-mentioned conditions for determining specific target individuals for each display unit 201. For example, the administrator sets specific target identification condition information for the display unit 201 set up in the region where the contracted organization is based, specifying that individuals who are presumed to be of interest or concern to the organization will be designated as specific target individuals. In this case, the content selection unit 104 selects content for each display unit 201.

[0037] The content stored in content server 4 is classified into content for specific target audiences and content for normal use. Specifically, content server 4 has a specific content storage section (not shown) and a normal content storage section (not shown). The specific content storage section stores content that is a candidate for being provided to specific target audiences, and the normal content storage section stores content for normal use. When the content selection unit 104 determines that the target person is a specific target person, it causes the reinforcement learning model to select content from the content stored in the specific content storage unit. If the content selection unit 104 determines that the target person is not a specific target person, it selects the normal content stored in the normal content storage unit. Furthermore, the content selection unit 104 may, for example, select the normal content when selecting the normal content, depending on the location where the display unit 201 is installed. In this case, the administrator or other person has set conditions (hereinafter referred to as "normal content selection conditions") in advance that define which content stored in the normal content storage unit should be used as the normal content for each display unit 201, and information indicating the normal content selection conditions (hereinafter referred to as "normal content selection condition information") is stored in a storage unit not shown. The administrator or other person can add or change the normal content selection condition information as appropriate.

[0038] The content selection unit 104 outputs information indicating the selected content (hereinafter referred to as "selected content information") to the content acquisition unit 105. The selected content information includes content ID information. For example, if content is selected for each display unit 201, the selected content information includes information that identifies the display unit 201 and information that associates the content ID with that information.

[0039] The content acquisition unit 105 acquires the content selected by the content selection unit 104 from the content server 4. In Embodiment 1, when the content acquisition unit 105 "acquires content," it means acquiring the content to be displayed from the content server 4. The content acquisition unit 105 outputs the acquired content to the display control unit 106.

[0040] The display control unit 106 causes the content acquired by the content acquisition unit 105 to be displayed on the display unit 201. More specifically, the display control unit 106 instructs the display unit 201 to display the content acquired by the content acquisition unit 105. When the display control unit 106 displays content on the display unit 201, it outputs information indicating that the content has been displayed (hereinafter referred to as "content displayed information") to the response information acquisition unit 107. The content displayed information includes information indicating that an instruction to display content on the display unit 201 has been output, and selected content information indicating the content ID of the content displayed on the display unit 201.

[0041] When the display control unit 106 displays content on the display unit 201, the reaction information acquisition unit 107 acquires information (hereinafter referred to as "reaction information") regarding the subject's reaction to the content displayed on the display unit 201. More specifically, the reaction information acquisition unit 107 acquires reaction information when it obtains content display information from the display control unit 106.

[0042] One example of a target audience's reaction to displayed content is that they pay close attention to the content. Furthermore, the above-mentioned reactions include, for example, the subject taking a picture of the content or reading a QR code (registered trademark; hereinafter omitted). Taking a picture of the content by the subject is intended to mean, for example, the subject taking an image of the screen of the display unit 201 on which the content is displayed using a mobile device such as a smartphone that they are holding. Reading a QR code by the subject is intended to mean, for example, the subject reading a QR code displayed as content using a mobile device such as a smartphone that they are holding. For example, the content may be a video displaying a QR code on which coupons, etc., are distributed. Another example of the above-mentioned reaction is the subject's facial expression. Another example of such a response is a touch operation of the display unit 201 by the subject. In this case, the display unit 201 is assumed to be a touch panel type digital signage.

[0043] The reaction information acquisition unit 107 acquires, for example, the captured image taken by the imaging unit 202, and acquires reaction information based on the acquired image. For example, the reaction information acquisition unit 107 performs known image recognition processing on the captured image to detect the subject's gaze direction. The reaction information acquisition unit 107 then calculates the time the subject's gaze is directed towards the content as the gaze time and acquires the calculated gaze time as reaction information. The reaction information acquisition unit 107 can store, for example, the captured image acquired from the imaging unit 202 or the information indicating the subject's gaze direction detected from the captured image in a memory unit that is not shown in chronological order, and calculate the gaze time the subject's gaze is directed towards the content from the stored captured image or information indicating the subject's gaze direction. Since the positional relationship between the display unit 201 and the imaging unit 202 is known in advance, the reaction information acquisition unit 107 can detect whether the subject is looking at the content displayed on the display unit 201 based on the detected gaze direction.

[0044] Furthermore, the reaction information acquisition unit 107 may, for example, perform known image recognition processing on the captured image to detect the subject's facial expression and acquire information indicating the detected facial expression as reaction information.

[0045] Furthermore, the reaction information acquisition unit 107 may, for example, perform known image recognition processing on the captured image to detect whether the subject is pointing their mobile device towards the display unit 201 or whether the subject is in a position to photograph the display unit 201. If it detects that the subject is pointing their mobile device towards the display unit 201 or is in a position to photograph the display unit 201, it may determine that the subject is photographing the screen of the display unit 201 on which content is displayed using a mobile device such as a smartphone that the subject is holding, and acquire information as reaction information that the subject is photographing the screen of the display unit 201 on which content is displayed using a mobile device such as a smartphone that the subject is holding.

[0046] Furthermore, the response information acquisition unit 107 may, for example, acquire information from the website indicating whether or not the website embedded in the QR code displayed as content has been accessed. If information indicating that the website has been accessed is acquired, it may acquire information indicating that the QR code has been read as response information.

[0047] Furthermore, the response information acquisition unit 107 may acquire information from the display unit 201 indicating that a touch operation has been performed on the display unit 201, and acquire this as response information. For example, the display unit 201 may display content and buttons to fast forward, rewind, or pause the content, in which case the display unit 201 detects that the user has performed a button operation and acquires information indicating that the button operation has been performed as response information. Also, for example, the display unit 201 may display a switch button for instructing a content switch. This switch button may, for example, be a switch button for switching the currently displayed content to content other than the current content, or, for example, if the content consists of multiple consecutive videos with different content, it may be a switch button for switching the video to be displayed within the content. Regarding the switching button when the content consists of multiple consecutive videos with different content, to give a specific example, suppose the orientation estimation unit 103 estimates that sports team B is the subject of interest or attention of the subject, and the display control unit 106 displays a video on the display unit 201 in which promotional videos of three popular players belonging to sports team B (let's call them player X, player Y, and player Z) are displayed in the order of player X's promotional video, player Y's promotional video, and player Z's promotional video (hereinafter referred to as "popular player videos"). In this case, for example, when displaying player X's promotional video, the display control unit 106 may also display a switching button to switch to player Y's promotional video and a switching button to switch to player Z's promotional video. In this case, the display unit 201 detects that the subject has operated a switching button and acquires information that the switching button has been operated as response information. The response information may include information indicating what kind of touch operation was performed.

[0048] The reaction information acquisition unit 107 outputs the acquired reaction information to the effect estimation unit 108 and the profile update unit 111. At this time, the reaction information acquisition unit 107 outputs the reaction information, along with the selected content information and orientation information, to the effect estimation unit 108 and the profile update unit 111. The reaction information acquisition unit 107 can acquire the selected content information, for example, from the content selection unit 104 via the content acquisition unit 105 and the display control unit 106. The reaction information acquisition unit 107 can also acquire the orientation information, for example, from the orientation estimation unit 103 via the content selection unit 104, the content acquisition unit 105, and the display control unit 106. Furthermore, if the response information acquisition unit 107 acquires information instructing a content switch as response information, for example, it may output information for instructing a content switch (hereinafter referred to as "content switch instruction information") to the content selection unit 104. Note that in Figure 1, the arrow from the response information acquisition unit 107 to the content selection unit 104 is not shown. In this case, the content selection unit 104 selects content upon receiving the output of the content switch instruction information. Then, the content acquisition unit 105 acquires the content selected by the content selection unit 104 from the content server 4, and the display control unit 106 displays the content acquired by the content acquisition unit 105 on the display unit 201. The content selection by the content selection unit 104, the content acquisition by the content acquisition unit 105, and the content display instruction by the display control unit 106 have already been explained, so redundant explanations are omitted. As a result, the information processing device 1 can switch the content to be displayed in response to direct instructions from the target user. Furthermore, for example, if the display unit 201 is displaying content consisting of multiple consecutive videos with different content, such as the example above where a popular player video and a switch button are displayed, and the reaction information acquisition unit 107 acquires information as reaction information that a switch button to display another video included in the content has been operated while one video included in the content is being displayed, the reaction information acquisition unit 107 may output information to the display control unit 106 to instruct the display to switch to the video corresponding to the operated switch button (hereinafter referred to as "content switching instruction information"). In this case, the display control unit 106, upon receiving the output of the content switching instruction information, switches the video within the content displayed on the display unit 201. In the example above, if a switch instruction to a promotional video of player Y is acquired as reaction information while a promotional video of player X is being displayed, the reaction information acquisition unit 107 outputs content switching instruction information to the display control unit 106 to instruct the display to switch to the promotional video of player Y. The display control unit 106 then instructs the display unit 201 to display the promotional video for player Y, even if the promotional video for player X is currently being displayed.

[0049] The effect estimation unit 108 estimates the effect of the content displayed on the display unit 201 by the display control unit 106. Specifically, the effect estimation unit 108 calculates the reward to be given to the reinforcement learning model based on the reaction information acquired by the reaction information acquisition unit 107 and the reward conditions. The reward condition defines how much reward the reinforcement learning model that displayed the content will receive, depending on how the target user reacts to the displayed content. The reward conditions are set in advance by the administrator or other designated personnel. Once the administrator or other designated personnel sets the reward conditions, they store information indicating the set reward conditions (hereinafter referred to as "reward condition information") in a memory unit (not shown). The administrator or other designated personnel can update the reward condition information as needed.

[0050] In the reward system, for example, a condition is set where the reward increases the longer the attention time. The reward is expressed, for example, as a score out of 10 points. This is merely one example, and the reward conditions could include, for example, a condition where a higher reward is calculated for positive expressions such as a smile or an expression of surprise; a condition where a higher reward is calculated when the subject takes a picture of the screen of the display unit 201 where the content is displayed, or when the QR code is scanned; or a condition where the reward increases or decreases depending on the operation performed when the display unit 201 is touched. Furthermore, the reward system may include conditions such as setting a lower reward for negative facial expressions, such as a frown. Note that what constitutes a positive or negative facial expression is predetermined by the administrator or other relevant personnel.

[0051] The effect estimation unit 108 outputs information indicating the calculated reward (hereinafter referred to as "reward information") to the model update unit 109. The effect estimation unit 108 outputs reward information, along with selected content information and orientation information, to the model update unit 109. The effect estimation unit 108 only needs to output the selected content information and orientation information output from the response information acquisition unit 107, along with the reward information, to the model update unit 109.

[0052] The model update unit 109 updates the reinforcement learning model based on orientation information, information indicating the content displayed on the display unit 201 (in other words, selected content information), and reward information. Specifically, the model update unit 109 updates the parameters of the reinforcement learning model. This causes the model update unit 109 to train the reinforcement learning model to select the optimal content based on the reward calculated by the effect estimation unit 108 based on the subject's response. When the reinforcement learning model is updated by the model update unit 109, the content selection unit 104 uses the updated reinforcement learning model to select content.

[0053] The model memory unit 110 stores the reinforcement learning model. In this example, the model storage unit 110 is assumed to be located within the information processing device 1, but this is merely one example. The model storage unit 110 may also be located outside the information processing device 1, in a location accessible to the information processing device 1.

[0054] The profile update unit 111 generates profile information about the subject. The profile update unit 111 then stores the generated profile information in the fan profile DB 5. The profile update unit 111 generates profile information (hereinafter referred to as "past response information") by associating the response information and selected content information output from the response information acquisition unit 107 with information used to identify the individual subject (hereinafter referred to as "personal identification information"). Furthermore, the profile update unit 111 may generate profile information by associating the orientation information output from the reaction information acquisition unit 107 with the object recognition information indicating a trademark or design detected by the object recognition unit 102 (hereinafter referred to as "past orientation information").

[0055] The above-mentioned personal identification information is assumed to be, for example, a facial image of the subject. For example, the profile update unit 111 can acquire the subject's facial image along with object recognition information from the object recognition unit 102. In this case, when the object recognition unit 102 recognizes, for example, an item owned or worn by the subject, it extracts the facial region in which the subject's face is captured from the captured image and outputs the image of that facial region as a facial image. Alternatively, for example, the profile update unit 111 may acquire the subject's facial image from the reaction information acquisition unit 107. In this case, the reaction information acquisition unit 107 extracts the facial region in which the subject's face is captured from the captured image acquired from the imaging unit 202 and outputs the image of that facial region as a facial image.

[0056] As described above, the target audience's interests or concerns are estimated, and it is not necessary for the target audience to be identified when content is displayed based on the estimation results. However, the profile update unit 111 generates profile information, and the generated profile information is stored in the fan profile DB5. For example, administrators can use the profile information as information for analyzing content previously displayed to individuals indicated by personally identifiable information, reactions to previously displayed content, and organizations or individuals previously estimated to be of interest or concern.

[0057] In this example, the profile update unit 111 is assumed to be located within the information processing device 1, but this is merely one example. The profile update unit 111 may be located outside the information processing device 1, in a location accessible to the information processing device 1.

[0058] The operation of the information processing device 1 according to Embodiment 1 will be described below. Figure 3 is a flowchart illustrating the operation of the information processing device 1 according to Embodiment 1. For example, when power is turned on to the information processing device 1, or when it receives an operation start instruction from an administrator, it starts the operation shown in the flowchart of Figure 3. The information processing device 1 repeats the operation shown in the flowchart of Figure 3 until, for example, the power to the information processing device 1 is turned off, or until it receives an operation end instruction from an administrator, etc. Administrators, for example, input a start command or an end command from a PC (Personal Computer) located in a control room or similar location that manages the information processing system 10. The control unit (not shown) of the information processing device 1 receives the start command and starts the operation of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, and profile update unit 111. The control unit also receives an end command and terminates the operation of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, and profile update unit 111.

[0059] In the following explanation of operation, we will use a specific example where a subject is wearing a uniform with the number "10" printed on it, belonging to a certain sports team B, and the subject is a specific individual. We will explain the operation of the information processing device 1 using an example where content is displayed to this subject. In this context, the criteria for determining a specific target person include at least the condition that "if the person of interest or concern is player X, then that person will be determined to be a specific target person." Furthermore, the reward conditions shall include at least a condition that rewards increase the longer the user is attentive to the displayed content.

[0060] The image acquisition unit 101 acquires the captured image of the subject from the imaging unit 202 (step ST10). The image acquisition unit 101 outputs the acquired image to the object recognition unit 102.

[0061] The object recognition unit 102 recognizes the subject's possessions or clothing based on the image acquired by the image acquisition unit 101 in step ST10, and detects any marks or designs included in the recognized possessions or clothing (step ST20). The object recognition unit 102 outputs object recognition information to the orientation estimation unit 103. Here, the object recognition unit 102 recognizes, for example, the uniform worn by the subject, and detects that the design of the uniform is that of sports team B, and that the number "10" is printed on the uniform. The object recognition unit 102 then outputs an image of sports team B's uniform, including the number "10," as object recognition information to the orientation estimation unit 103.

[0062] In step ST20, the orientation estimation unit 103 estimates the group or person that is of interest or concern to the subject based on the object recognition information output from the object recognition unit 102 (step ST30). The orientation estimation unit 103 outputs orientation information to the content selection unit 104. Here, the orientation estimation unit 103 estimates player X, who is the subject of the subject's interest or concern, based on, for example, an image of the uniform of sports team B that includes the character "10". The orientation estimation unit 103 then outputs information (for example, an image) indicating player X to the content selection unit 104.

[0063] The content selection unit 104 performs a content selection process (step ST40) to select content to provide from among the content related to the group or person stored in the content server 4, based on the orientation information indicating the group or person estimated by the orientation estimation unit 103 in step ST30.

[0064] In detail, the content selection unit 104 determines whether the target person is a specific target person or not, based on the estimation result of the orientation estimation unit 103, during the content selection process. If the content selection unit 104 determines that the target person is a specific target person, it selects content related to the group or person estimated by the orientation estimation unit 103 based on the orientation information. If the content selection unit 104 determines that the target person is not a specific target person, it selects the content that is normally set as content. The content selection unit 104 then outputs the selected content information to the content acquisition unit 105.

[0065] Here, first, the content selection unit 104 determines that the target person is a specific target person. Then, the content selection unit 104 selects, for example, a promotional video of player X. This promotional video is, for example, a video that sequentially shows a compilation of player X's plays, practice scenes, and footage of him participating in local events. The content selection unit 104 outputs selected content information, including the content ID of player X's promotional video, to the content acquisition unit 105.

[0066] The content acquisition unit 105 acquires the content selected by the content selection unit 104 from the content server 4 (step ST50). Here, the content acquisition unit 105 acquires a promotional video of player X from the content server 4. The content acquisition unit 105 outputs the acquired content to the display control unit 106.

[0067] The display control unit 106 displays the content acquired by the content acquisition unit 105 in step ST50 on the display unit 201 (step ST60). Here, the display control unit 106 causes the display unit 201 to display a promotional video of player X.

[0068] In step ST60, when the display control unit 106 displays content on the display unit 201, the reaction information acquisition unit 107 acquires reaction information (step ST70). The reaction information acquisition unit 107 outputs the acquired reaction information to the effect estimation unit 108 and the profile update unit 111. The reaction information acquisition unit 107 outputs the reaction information, along with the selected content information and orientation information, to the effect estimation unit 108 and the profile update unit 111. Here, the reaction information acquisition unit 107 acquires, for example, the amount of time the subject is looking at the content as reaction information. Now, suppose the subject stops and looks at the content for only 1 second. In this case, the reaction information acquisition unit 107 acquires reaction information indicating a look time of 1 second. The reaction information acquisition unit 107 then outputs the reaction information, along with the selected content information and orientation information, to the effect estimation unit 108 and the profile update unit 111.

[0069] The effect estimation unit 108 estimates the effect of the content displayed on the display unit 201 by the display control unit 106 in step ST60. Specifically, in step ST70, the effect estimation unit 108 calculates a reward based on the reaction information acquired by the reaction information acquisition unit 107 and the reward conditions (step ST80). The effect estimation unit 108 outputs reward information to the model update unit 109. The effect estimation unit 108 outputs the reward information, along with the selected content information and preference information, to the model update unit 109. Here, the effect estimation unit 108 calculates, for example, a low reward (for example, 1 out of 10 points) and outputs reward information indicating that low reward, along with selected content information and preference information, to the model update unit 109.

[0070] The model update unit 109 updates the reinforcement learning model based on orientation information, selected content information, and reward information (step ST90). In this case, because a low reward was given, the reinforcement learning model learns that when orientation information indicating player X, who belongs to sports team B, is input, the selection rate of player X's promotional videos decreases, and the selection rate of content other than player X's promotional videos increases.

[0071] The profile update unit 111 generates profile information about the subject (step ST100). Then, the profile update unit 111 stores the generated profile information in the fan profile DB5. Here, the profile update unit 111 generates profile information that associates, for example, reaction information indicating a gaze time of 1 second, selected content information including the content ID of player X's promotional video, and the subject's face image, and stores the generated profile information in the fan profile DB 5. In other words, the fan profile DB 5 stores profile information indicating that the subject, indicated by the face image stored in association with the promotional video of player X displayed on the display unit 201, gazed at the promotional video for only 1 second.

[0072] In the above explanation of the operation, we used as a specific example an example where a target person is wearing a uniform with the number "10" printed on it, belonging to a certain sports team B, and this target person is a specific target person, and the content is displayed to this target person, but this is only one example.

[0073] For example, suppose there is a subject who owns a bag embroidered with the logo of a certain school C, and school C is the entity responsible for the subject activity, and that subject is a specific subject. In this case, in step ST20, the object recognition unit 102 recognizes the bag, which is owned by the subject, and detects the School C logo embroidered on the recognized bag. The object recognition unit 102 then outputs the image of the bag, including the School C logo, as object recognition information to the orientation estimation unit 103. In step ST30, the orientation estimation unit 103 estimates that school C is the object of the subject's interest or concern based on an image of a bag that includes the logo of school C. The orientation estimation unit 103 then outputs orientation information indicating school C to the content selection unit 104. In step ST40, the content selection unit 104 provides orientation information to the reinforcement learning model and selects content by obtaining content IDs from the reinforcement learning model for a video announcing a club activity competition held at school C, a video soliciting donations to school C, or a video of a message of gratitude for support or encouragement to school C. In step ST40, the content acquisition unit 105 acquires from the content server 4 a video announcing a club activity competition held at school C, a video soliciting donations to school C, or a video of a message of gratitude for support or encouragement for school C. In step ST50, the display control unit 106 causes the display unit 201 to display a video announcing a club activity competition held at school C, a video soliciting donations to school C, or a video message of gratitude for support or encouragement given to school C.

[0074] Furthermore, the above explanation of operation assumed that the target person was a specific target person. However, if the target person is not a specific target person, in step ST40, the content selection unit 104 will select the content set as the normal content, in step ST50, the content acquisition unit 105 will acquire the normal content, and in step ST60, the display control unit 106 will display the normal content on the display unit 201. For example, a subject may not own or wear anything containing a trademark or design, or a subject may own or wear something containing a specific trademark or design, but whose interests or concerns are based on the organization or person inferred from that item are not considered a specific subject. Also, for example, a local government's informational video may be stored as regular content in content server 4, or more specifically, in the regular content storage section of content server 4. In this case, the content selection unit 104 selects the content ID of the local government's guidance video, which is the content for normal use (step ST40). Then, the content acquisition unit 105 acquires the local government's guidance video (step ST50), and the display control unit 106 displays the local government's guidance video on the display unit 201 (step ST60).

[0075] Furthermore, for example, if the response information acquisition unit 107 acquires information instructing content switching as response information in step ST70, the content switching instruction information is output to the content selection unit 104, and the operation of the information processing device 1 may be such that, for example, the processing of steps ST40 to ST60 is repeated in parallel with the processing of steps ST80 to ST100, or before the processing of step ST80.

[0076] Note that in the flowchart shown in Figure 3, the processing in step ST100 is assumed to occur after the processing in step ST90, but this is just one example. For example, the processing in step ST100 may occur before the processing in step ST80, or it may occur in parallel with the processing in steps ST80 to ST90. The processing in step ST100 only needs to occur after the processing in step ST70.

[0077] Furthermore, if the information processing device 1 is configured not to include a profile update unit 111, the information processing device 1 can omit the processing of step ST100 in the operation shown in the flowchart of Figure 3.

[0078] As described above, the information processing device 1 according to Embodiment 1 recognizes the subject's possessions or clothing based on the captured image of the subject, and detects trademarks or designs contained in the possessions or clothing. Based on the object recognition information indicating the detected trademarks or designs, the information processing device 1 estimates the group or person that is the subject's interest or concern, and based on the orientation information indicating the estimated group or person, selects content to provide from the content related to the group or person stored in the content server 4. Then, the information processing device 1 retrieves the selected content from the content server 4 and displays the retrieved content on the display unit 201. As a result, the information processing device 1 can provide information to a target person in the form of information provision using a display device, tailored to the group or person that the target person is interested in. The information processing device 1 can provide the target person with content that is highly likely to attract their interest.

[0079] In the above embodiment 1, the object recognition unit 102 may exclude people smaller than a preset size from the target if a person is present in the captured image. People who appear small in the captured image are assumed to be located far from the display unit 201.

[0080] Furthermore, in the above embodiment 1, the content selection unit 104 determined whether the subject was a specific target based on the estimation result of the group or person by the orientation estimation unit 103, and if it determined that the subject was a specific target, it selected content related to the group or person estimated by the orientation estimation unit 103 based on the orientation information. However, the content selection unit 104 does not necessarily have to determine whether the subject is a specific target. For example, if the information output device 2 is installed in a place where it is assumed that only specific targets exist, the content selection unit 104 may treat all subjects as specific targets and select content related to the group or person estimated by the orientation estimation unit 103 based on the orientation information.

[0081] <Example (1)> As described above, the profile update unit 111 generates past response information, which associates personal identification information with response information and selected content information, or past orientation information, which associates personal identification information with orientation information and object recognition information, as profile information and stores it in the fan profile DB5. In addition to the above-mentioned past response information or past orientation information, various other information can be managed as profile information in the Fan Profile DB5. For example, in Fan Profile DB5, information about an individual's past activities related to a group or person can be managed as profile information. Activities related to a group or person include, for example, purchasing merchandise or other related goods of the group or person, or participating in matches or events related to the group or person. In other words, in Fan Profile DB5, information related to the above activities can be managed as profile information, such as information about an individual's purchase history of merchandise such as goods from a group or person (hereinafter referred to as "merchandise purchase history information"), attendance history at match venues or event venues, or information about the purchase history of tickets for matches or events (hereinafter referred to as "participation history information"). Merchandise purchase history information is, for example, information that associates personal identification information with information indicating the goods or items purchased by the organization or person (e.g., product number), information indicating the date of purchase, and information indicating the place of purchase. Participation history information is, for example, information that associates personally identifiable information with information about the venue attended or tickets purchased (for example, information indicating the date of the match or event), information indicating the date of attendance or purchase, and information indicating the place of ticket purchase. Purchase history information or participation history information is generated, for example, by the profile update unit 111 and stored in the fan profile DB 5.

[0082] For example, the profile update unit 111 retrieves sales information from a merchandise management system (not shown) that manages merchandise sales by an organization or individual. The merchandise management system collects and manages sales information, also known as POS data, recorded when products are sold at stores, etc. The sales information includes information indicating who purchased which product, where, and how. The profile update unit 111 generates product purchase history information, which is the purchase history of each individual's merchandise, from the acquired sales information. For example, if the personal identification information is a facial image, but the sales information includes an ID or other information instead of a facial image as personal identification information, the profile update unit 111 can refer to the personal authentication database (not shown) to obtain the facial image corresponding to the ID. The personal authentication database is, for example, provided in the product sales system and stores personal authentication information in which IDs and facial images are associated.

[0083] For example, the profile update unit 111 retrieves attendance information from an event management system (not shown) that manages matches or events for a group or individual. Attendance information includes admission information or ticket purchase information. The event management system manages admission information recorded when entering a match or event venue, or ticket purchase information recorded when purchasing tickets. Admission information includes information indicating who entered which match or event. Ticket purchase information includes information indicating who purchased tickets for which match or event. The profile update unit 111 generates participation history information, which is each individual's entry or ticket purchase history, from the acquired entry information or ticket purchase information. For example, if the personal identification information is a facial image, but the entry information or ticket purchase information includes an ID or other information instead of a facial image as personal identification information, the profile update unit 111 can refer to the personal authentication DB and obtain the facial image corresponding to the ID.

[0084] Thus, in addition to past response information or past orientation information, the fan profile DB5 may also manage other profile information, such as merchandise purchase history information or participation history information.

[0085] Furthermore, in the above embodiment 1, if the object recognition unit 102 is unable to detect a trademark or design based on the captured image, the orientation estimation unit 103 may estimate the group or person that is the subject of the subject's interest or concern from profile information including information about trademarks or designs previously detected for the subject (past orientation information), information about content previously displayed to the subject (past reaction information), or information about the subject's past activities related to a group or person (product purchase history information or participation history information).

[0086] For example, suppose on January 10th, the information processing device 1 estimates that a certain subject (hereinafter referred to as "Subject 1") wearing the uniform of sports team B is interested in sports team B, and displays a video on the display unit 201 showing promotional videos of three popular players (players X, Y, and Z) belonging to sports team B in sequence. Sports team B is the entity responsible for the activity, and Subject 1 is a specific subject. On this day, Subject 1 is wearing the aforementioned uniform to watch a game of Sports Team B, and has already purchased a ticket for the game. In other words, as of January 10th, the Fan Profile DB5 manages profile information such as past reaction information showing that a video of a popular player was displayed to Subject 1 and their reaction to it, past orientation information indicating that Subject 1's interests or concerns were estimated to be related to Sports Team B, or participation history information indicating that Subject 1 purchased a ticket for Sports Team B's game held on January 10th.

[0087] Subsequently, on January 25th, the first subject was again imaged by the imaging unit 202. However, at this time, the first subject was not wearing the uniform mentioned above, but was in regular clothes. Furthermore, the object recognition unit 102 in the information processing device 1 was unable to detect the trademark or design. In the modified version (1), the orientation estimation unit 103 in the information processing device 1 may estimate the group or person that is the object of interest or concern of the first subject from past response information, past orientation information, or participation history information stored as profile information in the fan profile DB5.

[0088] For example, the orientation estimation unit 103 obtains the face image of the first subject, which was extracted by the object recognition unit 102 based on the image captured by the image acquisition unit 101 on January 25, from the object recognition unit 102. The orientation estimation unit 103 then matches the face image of the first subject obtained from the object recognition unit 102 with the personal identification information contained in the past orientation information stored as profile information in the fan profile DB5 to identify the past orientation information of the first subject, and from the identified past orientation information, estimates that sports team B is the object of interest or concern of the first subject. In this case, it is assumed that the personal identification information is a facial image. The orientation estimation unit 103 can use known image matching techniques to identify the personal identification information in which a person who is presumed to be the same person as the first subject is captured on the facial image of the first subject obtained from the object recognition unit 102.

[0089] Furthermore, for example, the orientation estimation unit 103 may compare the facial image of the first subject obtained from the object recognition unit 102 with past reaction information stored as profile information in the fan profile DB5. If the unit determines from the past reaction information that the first subject paused a video of a popular player when a promotional video of player Y was displayed, or that the screen was captured by a mobile device such as a smartphone when a promotional video of player Y was displayed, the unit may estimate that player Y is the object of interest or concern of the first subject.

[0090] Furthermore, for example, the orientation estimation unit 103 may compare the facial image of the first subject obtained from the object recognition unit 102 with past reaction information stored as profile information in the fan profile DB5. If it is estimated from the past reaction information that the first subject did not react well to videos of popular athletes, the unit may estimate that the athletes belonging to sports team B, excluding athletes X, Y, and Z, are the objects of interest or concern of the first subject.

[0091] Alternatively, for example, the orientation estimation unit 103 may compare the facial image of the first subject obtained from the object recognition unit 102 with information identifying the individual contained in the participation history information stored as profile information in the fan profile DB5, and from the participation history information, identify that the first subject watched a game of sports team B, and estimate that sports team B is the object of the first subject's interest or concern.

[0092] As a result, even if the subject was not wearing the uniform described above on January 25, the information processing device 1 can estimate what the subject is presumed to be interested in, and can provide content related to the estimated subject's interests.

[0093] The processing of the orientation estimation unit 103 as described in this <modification (1)> is performed in step ST30 of the operation of the information processing device 1 as described using the flowchart in Figure 3.

[0094] Thus, in the information processing device 1, if the object recognition unit 102 is unable to detect a trademark or design, the orientation estimation unit 103 may estimate the group or person that is the subject of the subject's interest or concern from profile information that includes information about trademarks or designs previously detected for the subject, information about content previously displayed to the subject, or information about past activities of the subject related to a group or person. As a result, if the information processing device 1 is unable to detect a trademark or design from the subject's possessions or clothing at present, it can estimate the group or person that is of interest or concern to the subject based on trademarks or designs previously detected from the subject's possessions or clothing, or from the subject's past activities.

[0095] <Example (2)> In the above embodiment 1, the information processing device 1, when it determined that the target person was a specific target person, selected content related to an organization or person based on orientation information. However, this is just one example, and for instance, the content selection unit 104 may select content using other information in addition to preference information. For example, the content selection unit 104 may select content based on the gender of the target audience, the age of the target audience, or information indicating the relationships between multiple target audiences (hereinafter referred to as "attribute information").

[0096] In this case, an example of the configuration of the information processing device 1a is as shown in Figure 4. Note that in Figure 4, for convenience, only the information processing device 1a is shown, but like the information processing device 1, the information processing device 1a is connected via a network to the information output device 2, the activity entity DB 3, the content server 4, and the fan profile DB 5. The information processing device 1a and the information output device 2 constitute the information processing system 10. In the example configuration of the information processing device 1a shown in Figure 4, the same reference numerals are used for components that are the same as those in the example configuration of the information processing device 1 already explained using Figure 1, and redundant explanations are omitted. The configuration example of the information processing device 1a differs from the configuration example of the information processing device 1 in that it includes an attribute determination unit 112.

[0097] The attribute determination unit 112 determines the attributes of the subject based on the captured image acquired by the image acquisition unit 101. In Embodiment 1, "attributes" include, for example, gender, age, or whether or not a relationship with other subjects is recognized. Age may be expressed as, for example, adult or child. In the information processing device 1a, the image acquisition unit 101 outputs the captured image to the object recognition unit 102 and the attribute determination unit 112.

[0098] The attribute determination unit 112 determines the gender or age of the subject based on the captured image, for example, using known image recognition technology. This is just one example, and the attribute determination unit 112 may determine the gender or age of the subject based on the captured image using other methods. The attribute determination unit 112, when multiple subjects are captured in the captured image, determines whether a relationship exists between the multiple subjects according to the relationship determination conditions. Relationship determination conditions are defined conditions that specify the state in which multiple people exist on an captured image in which a relationship between those people is determined. Relationship determination conditions are set in advance by the administrator or other designated person. When the administrator or other designated person sets the relationship determination conditions, they store information indicating the relationship determination conditions (hereinafter referred to as "relationship determination condition information") in a memory unit not shown in the diagram. The administrator or other designated person can update the relationship determination condition information as needed.

[0099] The relationship determination criteria may include, for example, the condition "If multiple subjects are standing close together, it is determined that a relationship exists between those subjects." Alternatively, the relationship determination criteria may also include, for example, the condition "If multiple subjects are conversing, it is determined that a relationship exists between those subjects." The attribute determination unit 112 determines, for example, whether there are multiple subjects standing close together in the captured image using an appropriate method. The conditions for determining what constitutes "standing close together" in the captured image are defined in the relationship determination conditions. Furthermore, the attribute determination unit 112 determines, for example, that multiple subjects in the captured image are conversing if the degree of aperture of multiple subjects in the captured image is equal to or greater than a preset threshold. The attribute determination unit 112 may determine the degree of aperture of subjects in the captured image using an appropriate method using known technology.

[0100] The attribute determination unit 112 outputs information indicating the attributes of the determined subject (hereinafter referred to as "attribute information") to the content selection unit 104. In attribute information, information identifying the subject is associated with the subject's gender, age, or information identifying subjects with whom a relationship is determined to exist. The information identifying the subject may, for example, be information indicating the subject's position in the captured image, or it may be an captured image from which the subject's face region has been extracted. Note that the information identifying the subject here does not have to be information that identifies the individual subject.

[0101] In this case, the content selection unit 104, based on the orientation estimation result of the orientation estimation unit 103, determines that the user is a specific target person and selects content related to the organization or person estimated by the orientation estimation unit 103, based on orientation information and attribute information.

[0102] For example, suppose the orientation estimation unit 103 estimates that sports team A is of interest to the subject (hereinafter referred to as "second subject"). Also, suppose the attribute determination unit 112 determines that the second subject is female. In this case, the content selection unit 104 provides the reinforcement learning model with orientation information indicating that the organization or person estimated to be of interest to the second subject is sports team A, and attribute information indicating that the second subject is female. The content selection unit 104 then obtains the content ID of a promotional video for a women's event on the day of a sports team A match where the women's event is being held, and selects the content from the reinforcement learning model. The content acquisition unit 105 then acquires the announcement video for the women's event from the content server 4, and the display control unit 106 displays the announcement video for the women's event on the display unit 201.

[0103] For example, suppose the orientation estimation unit 103 estimates that sports team A is the object of interest or concern of the subject (hereinafter referred to as "third subject"). Also, suppose the attribute determination unit 112 determines that the third subject is a child. In this case, the content selection unit 104 provides the reinforcement learning model with orientation information indicating that the organization or person estimated to be the object of interest or concern of the third subject is sports team A, and attribute information indicating that the third subject is a child. The content selection unit 104 then obtains the content ID of a video of sports team A's mascot playing rock-paper-scissors from the reinforcement learning model and selects the content. The content acquisition unit 105 then acquires the rock-paper-scissors video from the content server 4, and the display control unit 106 displays the rock-paper-scissors video on the display unit 201. In this case, for example, after a rock-paper-scissors video is displayed, the reaction information acquisition unit 107 may acquire an image from the imaging unit 202, detect the shape of the hand made by the third subject, and acquire as reaction information whether the third subject made rock, paper, or scissors. The reaction information acquisition unit 107 then outputs this reaction information to the display control unit 106, which determines whether the third subject won at rock-paper-scissors. If it determines that the third subject won, it may cause the display unit 201 to display a QR code that will emit light for a coupon. Note that in Figure 4, the arrow from the reaction information acquisition unit 107 to the display control unit 106 is not shown. Furthermore, this mechanism, in which the display control unit 106 determines whether there is any content to be displayed in addition to the already displayed content based on the reaction information acquired by the reaction information acquisition unit 107 and displays it on the display unit 201, can also be applied to the information processing device 1 as explained using Figure 1.

[0104] For example, suppose the orientation estimation unit 103 estimates that sports team A is the object of interest or concern of the subject (hereinafter referred to as "the fourth subject"). Also, suppose the attribute determination unit 112 recognizes that the fourth subject has a relationship with a subject in a similar position (hereinafter referred to as "the fifth subject"), and determines that the fourth subject is an adult and the fifth subject is a child. In this case, the content selection unit 104 provides the reinforcement learning model with orientation information indicating that the organization or person estimated to be the object of interest or concern of the fourth subject is sports team A, and attribute information indicating that a relationship exists between the fourth subject, who is an adult, and the fifth subject, who is a child. The content selection unit 104 then obtains the content ID of a video announcing an event for parents and children hosted by sports team A from the reinforcement learning model and selects the content. The content acquisition unit 105 then acquires the announcement video for the parent-child event from the content server 4, and the display control unit 106 displays the announcement video for the parent-child event on the display unit 201. In this way, when the information processing device 1a determines a combination of adults and children who are recognized to have a relationship, it can take this into consideration and display parent-child-oriented content, such as event announcements for sports team A that can be enjoyed by parents and children.

[0105] Furthermore, in this <modification (2)>, the profile update unit 111 in the information processing device 1a may generate information regarding the attributes of the subject (hereinafter referred to as "content provision attribute information") as profile information, and store the generated content provision attribute information as profile information in the fan profile DB 5. Content provision attribute information is information to which personal identification information and information indicating the attributes of the subject are associated. For example, if the attribute of the determined subject is an attribute that indicates a relationship with other subjects, personal identification information of the other subjects may be included in the content provision attribute information. The profile update unit 111 acquires attribute information from the attribute determination unit 112 via the content selection unit 104, the content acquisition unit 105, and the display control unit 106, and uses the acquired attribute information as information indicating the attributes of the target person included in the attribute information at the time of content provision.

[0106] Figure 5 is a flowchart illustrating the operation of the information processing device 1a, which is equipped with an attribute determination unit 112, in Embodiment 1. In the flowchart of Figure 5, the specific operations of the information processing device 1a during steps ST10, ST20, and ST40-ST100 are the same as those of the information processing device 1 during steps ST10, ST20, and ST40-ST100 in the flowchart of Figure 3, which have already been explained, so redundant explanations are omitted.

[0107] In step ST25, the attribute determination unit 112 determines the attributes of the subject based on the captured image acquired by the image acquisition unit 101. The attribute determination unit 112 then outputs the attribute information to the content selection unit 104.

[0108] Based on the estimation results of the orientation estimation unit 103 for a group or person, the content selection unit 104 determines that the person is a specific target person and selects content related to the group or person estimated by the orientation estimation unit 103, based on orientation information and attribute information (step ST30a).

[0109] Note that in the flowchart shown in Figure 5, the processes of step ST20 and step ST25 are assumed to be performed in parallel, but this is just one example. For example, the process of step ST25 may be performed after the process of step ST20, or vice versa. It is sufficient that the processes of step ST20 and step ST25 are completed before the process of step ST30a is performed.

[0110] Furthermore, if the information processing device 1a is configured not to include the profile update unit 111, the information processing device 1a can omit the processing of step ST100 in the operation shown in the flowchart of Figure 5.

[0111] Thus, the information processing device 1a may be configured to include an attribute determination unit 112 that determines the attributes of the subject, including the subject's gender, age, or whether or not there is a relationship with other subjects, based on the captured image acquired by the image acquisition unit 101, and the content selection unit 104 may be configured to select content based on orientation information and attribute information indicating attributes. As a result, the information processing device 1a can provide information to the target person in a more appropriate manner, taking into account the target person's attributes, that is tailored to the organizations or individuals that the target person is interested in or concerned with.

[0112] <Variation (3)> In the above embodiment 1, when the information processing device 1 displays content on the display unit 201, it obtains the subject's reaction to the displayed content from the actions taken spontaneously by the subject who viewed the content. This is just one example; for instance, the information processing device 1b (see Figure 6, described later) may display content on the display unit 201, ask questions to the subject, and obtain the subject's response to the questions, thereby obtaining the subject's reaction to the displayed content.

[0113] In this case, an example of the configuration of the information processing device 1b would be, for example, the configuration shown in Figure 6. Note that, for convenience, only the information processing device 1b is shown in Figure 6. However, like the information processing device 1, the information processing device 1b is connected via a network to the information output device 2, the activity entity DB 3, the content server 4, and the fan profile DB 5. The information processing device 1b and the information output device 2 constitute the information processing system 10. In the example configuration of the information processing device 1b shown in Figure 6, the same reference numerals are used for components that are the same as those in the example configuration of the information processing device 1 already explained using Figure 1, and redundant explanations are omitted. The configuration example of the information processing device 1b differs from the configuration example of the information processing device 1 in that it includes a questioning unit 113.

[0114] When the display control unit 106 displays content on the display unit 201, the questioning unit 113 asks questions to obtain the subject's response to the displayed content. More specifically, when the questioning unit 113 obtains content display information from the display control unit 106, it generates questions to obtain the subject's response to the displayed content. In the information processing device 1b, the display control unit 106 outputs the content display information to the questioning unit 113. The question unit 113 generates questions based on, for example, the displayed content information, according to the question generation conditions. The question generation conditions are defined as conditions that specify what kind of questions should be generated when certain content is displayed. The question generation conditions are generated in advance by an administrator or the like, and information indicating the question generation conditions (hereinafter referred to as "question generation condition information") is stored in a storage unit (not shown). The administrator or the like can update the question generation condition information as needed. This is merely one example, and the question unit 113 may generate questions using other methods. For example, the question unit 113 may generate questions using a machine learning model. The machine learning model here is assumed to be a model trained through supervised learning, which takes displayed content information as input and outputs a question. This machine learning model is generated in advance and stored, for example, in the model storage unit 110.

[0115] For example, suppose the preference estimation unit 103 estimates that sports team B is of interest to the subject, and the display control unit 106 displays a video of popular players on the display unit 201, showing promotional videos of three popular players (let's call them player X, player Y, and player Z) belonging to sports team B in order. In this case, the question unit 113 receives this and generates the question, "Have you found your favorite player?" When the question unit 113 generates a question, it outputs information for outputting the question (hereinafter referred to as "question output control information") to, for example, the display unit 201 of the information output device 2. Here, the question output control information is assumed to be information for displaying the question.

[0116] The display unit 201 receives the question output control information and displays the question, "Have you found your favorite player?" The user views the above question and enters their answer via the display unit 201. The question unit 113 displays the question and an input area for entering an answer (for example, buttons labeled "Player X is my favorite," "Player Y is my favorite," and "Player Z is my favorite"). The type of input area to display is determined by the question unit 113 when the question is generated. When a participant answers by touching the input unit, the response information acquisition unit 107 acquires information from the display unit 201 indicating which answer was selected, and uses this as response information.

[0117] The reaction information acquisition unit 107 outputs the acquired reaction information to the effect estimation unit 108 and the profile update unit 111. The effect estimation unit 108 calculates rewards based on the responses from the subjects. In the example above, for instance, if a subject responds that "Player X is my favorite," a higher reward is calculated for content related to Player X, and lower rewards are calculated for content related to Players Y and Z. The model update unit 109 then trains the reinforcement learning model to prioritize selecting content related to player X when it is given orientation information indicating that sports team B is of interest to the subject. This improves the accuracy of content selection by the reinforcement learning model. The profile update unit 111 generates profile information by associating, for example, personal identification information of the subject, response information indicating that a response such as "Player X is my favorite" was received, and selected content information including the content ID of a popular player's video with the selected content information, and stores the generated profile information in the fan profile DB 5.

[0118] In the modified example (3), the questioning unit 113 does not necessarily have to ask questions to obtain the subject's reaction to the displayed content when the display control unit 106 displays content on the display unit 201. For example, the content to be questioned may be set in advance by an administrator, and the display control unit 106 may output content display information to the questioning unit 113 only when it displays content to be questioned on the display unit 201. If content that is not to be questioned is displayed on the display unit 201, the content display information may be output to the reaction information acquisition unit 107 and not to the questioning unit 113.

[0119] Figure 7 is a flowchart illustrating the operation of the information processing device 1b, which is equipped with a questioning unit 113, in Embodiment 1. In the flowchart of Figure 7, the specific operations of the information processing device 1b during steps ST10 to ST60 and steps ST70 to ST100 are the same as those of the information processing device 1 during steps ST10 to ST60 and steps ST70 to ST100 in the flowchart of Figure 3, which have already been explained, so redundant explanations are omitted.

[0120] When the display control unit 106 displays content on the display unit 201 in step ST60, the questioning unit 113 performs questioning processing to obtain the subject's response to the displayed content (step ST65). More specifically, when the questioning unit 113 obtains content display information from the display control unit 106, it generates questions to obtain the subject's response to the displayed content. In step ST60, the display control unit 106 outputs the content display information to the questioning unit 113.

[0121] Furthermore, if the information processing device 1b is configured not to include the profile update unit 111, the information processing device 1b can omit the processing of step ST100 in the operation shown in the flowchart of Figure 7.

[0122] Thus, the information processing device 1b may be configured such that when the display control unit 106 displays content on the display unit 201, it includes a questioning unit 113 that asks questions to obtain the subject's reaction to the displayed content, and the reaction information acquisition unit 107 acquires the answers to the questions asked by the questioning unit 113 as reaction information. This allows the information processing device 1b to further improve the accuracy of content selection when providing information to a target person that is tailored to the organizations or individuals that the target person is interested in.

[0123] <Example (4)> In the above embodiment 1, the information output device 2 was assumed to have the configuration described with reference to Figures 1 and 2. However, this is merely one example, and the information output device 2a (see Figures 8 and 9 described later) may also be configured to include, for example, a microphone 203 and a speaker 204. In this case, an example of the configuration of the information output device 2a would be as shown in Figure 8. The configuration example of the information output device 2a shown in Figure 8 differs from the configuration example of the information output device 2 shown in Figure 1 in that it includes a microphone 203 and a speaker 204.

[0124] Figure 9 is a diagram illustrating the general appearance of the information output device 2a as shown in Figure 8. Figure 9 shows the information output device 2a viewed from the front. As shown in Figure 9, the information output device 2a includes a main unit (indicated as "M" in Figure 9; not shown in Figure 8), a display unit 201 and an imaging unit 202 provided on the main unit, as well as a microphone 203 and a speaker 204 provided on the main unit. Since the display unit 201 and the imaging unit 202 have already been explained, we will omit any further explanation. Microphone 203 is a sound acquisition device. Speaker 204 is an audio output device.

[0125] In Figure 9, the main unit is shown as a housing, but the shape of the main unit is not specified. The information output device 2a only needs to be equipped with a display unit 201, an imaging unit 202, a microphone 203, and a speaker 204. Furthermore, in Figure 9, the microphone 203 and speaker 204 are shown to be located below the display unit 201 when viewed from the front with the information output device 2a installed, but this is merely one example. For example, the microphone 203 may be located above the display unit 201 when viewed from the front with the information output device 2a installed, or the speaker 204 may be located above the display unit 201 when viewed from the front with the information output device 2a installed. Furthermore, while Figure 9 shows that the information output device 2a is equipped with one microphone 203 and two speakers 204, this is merely an example, and the number of microphones 203 is not limited to one, nor is the number of speakers 204 limited to two.

[0126] In the modified version (4), the information processing device 1 may, for example, have the response information acquisition unit 107 acquire the spoken voice collected by the microphone 203 and acquire response information based on the acquired spoken voice. For example, the response information acquisition unit 107 performs known speech recognition processing on the spoken audio to recognize the content of the subject's speech. The response information acquisition unit 107 then acquires the recognized speech content as response information. In this case, the reward conditions may include, for example, a condition where a higher reward is calculated if positive keywords are spoken. Positive keywords are set in advance by the administrator or other relevant personnel. The administrator or other relevant personnel may set keywords such as "cool, looks cool, or was cool," "cute, or was cute," "funny, looks fun, or was fun," "want to go, want to go, or went," "amazing, incredible, or was amazing," "crazy, crazy, or was crazy," "good, looks good, or was good," "like," "my favorite," "what is this," "interested," "when, where, or who," "<organization name>," and "<personal name>" as positive keywords. Furthermore, the reward conditions may include, for example, a condition where a lower reward is calculated if negative keywords are spoken.

[0127] <Variation (5)> In the above embodiment 1, the information output device may be an information output device 2b (see Figures 10 and 11 described later) that includes a character unit 205, in addition to the configuration of the information output device 2a described in <Modification (4)> above. In this case, an example of the configuration of the information output device 2b is as shown in Figure 10. The configuration example of the information output device 2b shown in Figure 10 differs from the configuration example of the information output device 2a shown in Figure 8 in that it includes a character unit 205.

[0128] Figure 11 is a diagram illustrating the general appearance of the information output device 2b as shown in Figure 10. Figure 11 shows the information output device 2b viewed from the front. As shown in Figure 11, the information output device 2b comprises a main unit (indicated as "M" in Figure 11; not shown in Figure 10), a display unit 201 provided on the main unit, an imaging unit 202 provided on the main unit, a microphone 203 provided on the main unit, a speaker 204 provided on the main unit, and a character unit 205 provided on the main unit. Since the display unit 201, imaging unit 202, microphone 203, and speaker 204 have already been explained, we will omit any further explanation. Character category 205 refers to mascot dolls, etc., that represent a specific group or person.

[0129] In Figure 11, the main unit is shown as a housing, but the shape of the main unit is not specified. The information output device 2b only needs to be equipped with a display unit 201, an imaging unit 202, a microphone 203, a speaker 204, and a character unit 205. Furthermore, in Figure 11, the microphone 203 and speaker 204 are shown to be located below the display unit 201 when viewed from the front with the information output device 2b installed, but this is merely one example. For example, the microphone 203 may be located above the display unit 201 when viewed from the front with the information output device 2b installed, and the speaker 204 may be located above the display unit 201 when viewed from the front with the information output device 2b installed. Furthermore, while Figure 11 shows that the information output device 2b is equipped with one microphone 203 and two speakers 204, this is merely an example, and the number of microphones 203 is not limited to one, nor is the number of speakers 204 limited to two. Furthermore, in Figure 11, the character unit 205 is shown to be located next to the display unit 201 in the main unit with the information output device 2b installed, but this is merely one example, and the location of the character unit 205 is not limited to this. For example, the character unit 205 may be located above the display unit 201. For example, a microphone 203 and a speaker 204 may be provided on the character unit 205. This allows the information processing device 1 to give the target person the impression that they are interacting with the character unit 205.

[0130] In this document, we have described an information output device 2b that includes a character unit 205 in addition to the configuration of the information output device 2a described in <Modification (4)> above. However, this is merely one example, and for example, the information output device 2 shown in Figure 1 may also be further equipped with a character unit 205. In other words, the information output device may be configured to include a display unit 201, an imaging unit 202, and a character unit 205.

[0131] <Differentiation (6)> In the above embodiment 1, the information output device may be an information output device 2c (see Figures 12 and 13 described later) that includes a moving mechanism 206, in addition to the configuration of the information output device 2 described in Figure 1. In this case, an example of the configuration of the information output device 2c is as shown in Figure 12. The configuration example of the information output device 2c shown in Figure 12 differs from the configuration example of the information output device 2 shown in Figure 1 in that it includes a moving mechanism 206.

[0132] Figure 13 is a diagram illustrating the general appearance of the information output device 2c as shown in Figure 12. Figure 13 shows the information output device 2c viewed from the front. As shown in Figure 13, the information output device 2c comprises a main body (indicated as "M" in Figure 13; not shown in Figure 12), a display unit 201 and an imaging unit 202 provided on the main body, and a moving mechanism 206 provided on the main body. Since the display unit 201 and the imaging unit 202 have already been explained, we will omit any further explanation. The mobility mechanism 206 makes the information output device 2c movable. The mobility mechanism 206 is, for example, a caster or a tire. The mobility mechanism 206 is, for example, attached to the bottom of the body of the information output device 2c, and allows the information output device 2c to move in any direction.

[0133] For example, by making the information output device an information output device 2c equipped with a movable mechanism 206 as shown in Figure 12, administrators can freely change the installation location of the information output device 2c. For example, administrators may change the installation location to a place where a large number of specific target individuals are likely to be present. In addition to the configuration of the information output device 2 as described using Figure 1, an information output device 2c further equipped with a moving mechanism 206 has been described here. However, this is merely one example, and for example, the configuration of the information output devices 2a and 2c as described in <Modification (4)> or <Modification (5)> above may be further equipped with a moving mechanism 206.

[0134] Furthermore, in the modified example (6), the moving mechanism 206 may be composed of, for example, an electric motor and wheels. In this case, the information output device 2c can be a self-propelled information output device 2c. In this case, for example, in the information output device 2c, the moving mechanism 206 operates based on instructions from the position optimization unit 114 (see Figure 14, described later). As the moving mechanism 206 operates based on instructions from the position optimization unit 114, the information output device 2c moves autonomously. In this case, the information processing device 1c includes a position optimization unit 114, as shown in Figure 14, for example.

[0135] Figure 14 shows an example of the configuration of the information processing device 1c, which is equipped with a position optimization unit 114 in Embodiment 1. Note that, for convenience, only the information processing device 1c and the information output device 2c are shown in Figure 14. However, like the information processing device 1, the information processing device 1c is connected to the activity entity DB3, content server 4, and fan profile DB5 via a network. The information processing device 1c and the information output device 2c constitute the information processing system 10. In the example configuration of the information processing device 1c shown in Figure 14, the same reference numerals are used for components that are the same as those in the example configuration of the information processing device 1 already explained using Figure 1, and redundant explanations are omitted. The configuration example of the information processing device 1c differs from the configuration example of the information processing device 1 in that it includes a position optimization unit 114.

[0136] The position optimization unit 114 acquires captured images from the imaging unit 202 and measures pedestrian flow based on the acquired captured images. In Embodiment 1, "pedestrian flow" includes the number of subjects captured in the captured images, their orientation, or their movement speed. For example, the position optimization unit 114 stores the captured images acquired from the imaging unit 202 in a time-series memory unit (not shown), and periodically (for example, every second) analyzes the stored time-series consecutive captured images using known image recognition technology to measure the orientation or movement speed of the subjects. The position optimization unit 114 can measure the movement speed of the subjects using known technology that analyzes consecutive image frames to track the movement trajectory of a person.

[0137] The position optimization unit 114 then determines whether the information output device 2c needs to move, and if so, the direction and amount of movement, based on the measured pedestrian flow, i.e., the number of people, their orientation, or their movement speed.

[0138] For example, the position optimization unit 114 determines to move the information output device 2c if the number of subjects is below a preset threshold. Furthermore, for example, the position optimization unit 114 determines that if the subject is facing a direction other than the direction of the display unit 201, it should move the information output device 2c. Furthermore, for example, the position optimization unit 114 determines that the information output device 2c should be moved if the subject is moving at a speed exceeding a preset threshold. For example, the position optimization unit 114 may determine whether or not to move the information output device 2c by considering the reward calculated by the effect estimation unit 108. For example, even if the number of subjects is above a preset threshold, if the reward calculated by the effect estimation unit 108 is less than the preset threshold, it may decide to move the information output device 2c. In this case, the effect estimation unit 108 outputs the reward information to the position optimization unit 114.

[0139] Furthermore, when the position optimization unit 114 determines that the information output device 2c should be moved, it determines in which direction and by how much to move the information output device 2c, according to the number, orientation, or movement speed of the subjects estimated from the captured image. For example, the position optimization unit 114 determines in which direction and by how much to move the information output device 2c, based on the number, orientation, or movement speed of subjects estimated from the captured image, according to conditions set in advance by an administrator or the like (hereinafter referred to as "movement determination conditions"). For example, the position optimization unit 114 may determine to move the information output device 2 in any direction by any amount, or to move the information output device 2 in the direction the subject is facing by any amount, or to move the information output device 2 in the direction the subject is moving by an amount corresponding to the subject's movement speed. Furthermore, for example, the position optimization unit 114 may use a machine learning model to determine the direction and amount of movement of the information output device 2c. The machine learning model here is assumed to be a model trained through supervised learning, which takes information indicating the number, orientation, or movement speed of the subjects as input and outputs the direction and amount of movement of the information output device 2c.

[0140] When the position optimization unit 114 determines the direction and amount of movement of the information output device 2c, it generates information (hereinafter referred to as "position change information") for moving the information output device 2c in the determined direction and by the determined amount of movement, and outputs it to the movement mechanism 206 of the information output device 2c. The information output device 2c is equipped with various sensors (not shown). These sensors include GPS (Global Positioning System) sensors and gyro sensors, etc. The position optimization unit 114 can determine the current position of the information output device 2c from sensor information acquired from the various sensors of the information output device 2c, and can convert the direction and amount of movement to which the information output device 2c should be moved, determined based on the captured image, into the direction and amount of movement as seen from the information output device 2c. Alternatively, for example, information indicating the current installation position and orientation of the information output device 2c may be stored in a storage unit (not shown), and the position optimization unit 114 may refer to the storage unit to determine the current installation position of the information output device 2c.

[0141] When position change information is output from the position optimization unit 114, the movement mechanism 206 of the information output device 2c is driven according to the position change information. This makes it possible to improve the efficiency of information provision so that the information processing device 1c can be most effective in the optimal location.

[0142] Figure 15 is a flowchart illustrating the operation of the information processing device 1c, which is equipped with a position optimization unit 114 in Embodiment 1. In the flowchart of Figure 15, the specific operations of the information processing device 1c during steps ST10 to ST100 are the same as those of the information processing device 1 during steps ST10 to ST100 in the flowchart of Figure 3, which have already been explained, so redundant explanations are omitted.

[0143] The position optimization unit 114 acquires captured images from the imaging unit 202, measures the flow of people based on the acquired captured images, and determines whether the information output device 2c needs to move and, if so, in what direction, based on the measured flow of people. The position optimization unit 114 then generates position change information and outputs it to the movement mechanism 206 of the information output device 2c (step ST105).

[0144] In the flowchart of Figure 15, the operation of the information processing device 1c is shown to be performed after the processing of step ST100, but this is just one example, and the processing of step ST105 may be performed at an appropriate timing. Furthermore, if the information processing device 1c is configured not to include the profile update unit 111, the information processing device 1c can omit the processing of step ST100 in the operation shown in the flowchart of Figure 15.

[0145] Furthermore, in the above description of <Modification (6)>, a self-propelled information output device 2c equipped with a mobile mechanism 206 is assumed to be connected to the information processing device 1c. However, this is merely one example, and for example, an information output device 2 (see Figure 1), an information output device 2a (see Figure 8), or an information output device 2b (see Figure 10) without a mobile mechanism 206 may be connected to the information processing device 1c. In this case, for example, in the information processing device 1c, the position optimization unit 114 stores the position change information in a storage unit (not shown). For example, an administrator may display the position change information stored in the storage unit on a PC or the like to confirm it, and then change the installation position of the information output device 2, information output device 2a, or information output device 2b.

[0146] Thus, the information processing device 1c may be configured to include a position optimization unit 114 that measures pedestrian flow based on the captured image taken by the imaging unit 202 and generates position change information for changing the position of the information output device 2c based on the measured pedestrian flow. This makes it possible to improve the efficiency of information provision so that the information processing device 1c can be most effective in the optimal location.

[0147] Furthermore, in the above embodiment 1, multiple variations (1) to (6) described above may be applied in combination.

[0148] For example, in Embodiment 1, the above-described <Modification (1)> and <Modification (2)> may be applied in combination. In this case, for example, the information processing device includes, in addition to the configuration of the information processing device 1 as described with reference to Figure 1, an attribute determination unit 112 (see Figure 4) and a query unit 113 (see Figure 6).

[0149] Let's explain with a specific example. In the following explanation of the specific example, the information processing device 1 equipped with an attribute determination unit 112 and a query unit 113 will also be referred to as a "function-enhanced information processing device". For example, suppose on January 10th, the orientation estimation unit 103 of the function-enhanced information processing device estimates that a certain subject wearing the uniform of sports team B (hereinafter referred to as "subject 6") has an interest in sports team B. Also, suppose the attribute determination unit 112 of the function-enhanced information processing device determines that subject 6 has an attribute that indicates a relationship with another subject (hereinafter referred to as "subject 7") located close to subject 6 in the captured image. In the enhanced information processing device, the content selection unit 104 selects a popular player video, which displays promotional videos of three popular players (let's call them player X, player Y, and player Z) belonging to sports team B in order, and the display unit 201 then displays the popular player video. On this day, Subject 6 was wearing the aforementioned uniform to watch a sports team B game, while Subject 7 was not wearing a uniform and was in regular clothes. As of January 10th, the Fan Profile DB5 manages profile information such as past reaction information showing when popular player videos were displayed to the 6th subject and their reactions to them, past orientation information indicating that the 6th subject's interests or concerns were estimated to be related to sports team B, and content provision attribute information including the personally identifiable information of the 7th subject as information about subjects who are recognized to have a relationship with the 6th subject.

[0150] Subsequently, on January 25th, the seventh subject was again imaged by the imaging unit 202. At this time, the seventh subject was not wearing the uniform mentioned above, but was in regular clothes. Furthermore, the object recognition unit 102 of the function-enhanced information processing device was unable to detect the trademark or design. In this case, the functional information processing device may estimate the group or person that is the object of interest or concern of the seventh target person from the past response information, past orientation information, and content provision attribute information stored as profile information in the fan profile DB5.

[0151] For example, the orientation estimation unit 103 obtains the face image of the seventh subject, which was extracted by the object recognition unit 102 based on the image captured by the image acquisition unit 101 on January 25, from the object recognition unit 102. The orientation estimation unit 103 then compares the face image of the seventh subject obtained from the object recognition unit 102 with the content provision attribute information stored as profile information in the fan profile DB5 to identify that the seventh subject was previously determined to be a subject with a relationship to the sixth subject. Next, the orientation estimation unit 103 searches the fan profile DB5 for past orientation information or past response information stored as profile information to see if there is any information about the sixth subject. For example, from the past orientation information, it identifies that in the past, sports team B was estimated to be an object of interest or concern for the sixth subject. Based on this, the orientation estimation unit 103 may estimate that sports team B is an object of interest or concern for the seventh subject. On January 10, Subject 7, who was determined to have a relationship with Subject 6, is presumed to have been with Subject 6 on that day. In other words, Subject 7 is presumed to have been watching a game of Sports Team B and is presumed to have an interest in or concern for Sports Team B. Because the attribute information of Subject 7 at the time of content provision, including personally identifiable information, is stored as profile information for subjects who have a relationship with Subject 6, the functional information processing device can estimate the subjects that Subject 7 is presumed to be interested in or concerned with, even if Subject 7 was not wearing a uniform on January 25.

[0152] Furthermore, for example, in Embodiment 1, the above-described <Modification (3)> and <Modification (4)> may be applied in combination. In this case, for example, the information processing system 10 is composed of an information processing device 1b as described using Figure 6 and an information output device 2a as described using Figures 8 and 9.

[0153] In this case, in the information processing device 1b, when the question unit 113 generates a question, it may output question output control information to the speaker 204 of the information output device 2a so as to output the question as voice. Furthermore, in the information processing device 1b, the response information acquisition unit 107 may acquire the voice of the subject responding to the question voice output from the speaker 204 from the microphone 203.

[0154] Thus, in the information processing system 10, the information output device 2a has a microphone 203 and a speaker 204, and the information processing device 1b includes a question unit 113 that, when the display control unit 106 displays content on the display unit 201, outputs a question from the speaker 204 in voice to obtain the subject's response to the displayed content, and the response information acquisition unit 107 can be configured to acquire the spoken voice in response to the question collected by the microphone 203 as response information. This allows the information processing system 10 to provide information to the target audience in a more natural and intuitive user experience.

[0155] Furthermore, for example, in Embodiment 1, the above-described <Modification (3)> and <Modification (5)> may be applied in combination. In this case, for example, the information processing system 10 is composed of an information processing device 1b as described using Figure 6 and an information output device 2b as described using Figures 10 and 11. In this case, for example, in the information processing system 10, the question unit 113 of the information processing device 1b can output a question from the speaker 204 provided in the character unit 205 to obtain the subject's response to the displayed content when the display control unit 106 causes the display unit 201 to display content. As a result, the information processing system 10 can provide information to the target audience in a more natural and intuitive user experience, and can also provide such information in a more intimate way.

[0156] Figures 16A and 16B show an example of the hardware configuration of the information processing device 1 according to Embodiment 1. In Embodiment 1, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, and a control unit (not shown) are realized by the processing circuit 1001. That is, the information processing device 1 includes a processing circuit 1001 for recognizing the subject's possessions or clothing based on the acquired image, detecting a mark or design displayed on the recognized possessions or clothing, estimating a group or person that is the subject's interest or concern, and controlling the display unit 201 to display content related to the estimated group or person based on orientation information indicating the estimated group or person. The processing circuit 1001 may be dedicated hardware as shown in Figure 16A, or it may be a processor 1004 that executes a program stored in memory as shown in Figure 16B.

[0157] If the processing circuit 1001 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0158] When the processing circuit is a processor 1004, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, and a control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 reads and executes the program stored in memory 1005, thereby executing the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, and a control unit (not shown). In other words, the information processing device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, will result in the execution of steps ST10 to ST100 in Figure 3 described above. Furthermore, the program stored in the memory 1005 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, and the control unit (not shown). Here, memory 1005 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.

[0159] Furthermore, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, and the control unit (not shown) may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the image acquisition unit 101 and reaction information acquisition unit 107 can be implemented by a processing circuit 1001 as dedicated hardware, while the functions of the object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, effect estimation unit 108, model update unit 109, profile update unit 111, and the control unit (not shown) can be implemented by the processor 1004 reading and executing a program stored in memory 1005. The model storage unit 110 and the storage unit (not shown) consist of, for example, memory 1005, an HDD, or an SSD. Furthermore, the information processing device 1 includes information output devices 2, 2a, 2b, 2c, activity entity DB3, content server 4, and fan profile DB5, as well as an input interface device 1002 and an output interface device 1003 for wired or wireless communication.

[0160] If the information processing device is an information processing device 1a as shown in Figure 4, the hardware configuration of the information processing device 1a is the same as the configuration shown in Figures 16A and 16B. In the information processing device 1a, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, attribute determination unit 112, and a control unit (not shown) are realized by the processing circuit 1001. In other words, the information processing device 1a includes a processing circuit 1001 for controlling content selection based on attribute information in addition to orientation information. The processing circuit 1001 may be dedicated hardware as shown in Figure 16A, or it may be a processor 1004 that executes a program stored in memory as shown in Figure 16B.

[0161] The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the attribute determination unit 112, and a control unit (not shown). In other words, the information processing device 1a includes a memory 1005 for storing a program that, when executed by the processing circuit 1001, will result in the execution of steps ST10 to ST100 in Figure 5 described above. Furthermore, the program stored in memory 1005 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the attribute determination unit 112, and the control unit (not shown). Furthermore, the information processing device 1a includes information output devices 2, 2a, 2b, 2c, activity entity DB3, content server 4, and fan profile DB5, as well as an input interface device 1002 and an output interface device 1003 for wired or wireless communication.

[0162] If the information processing device is the information processing device 1b shown in Figure 6, the hardware configuration of the information processing device 1b is also the configuration shown using Figures 16A and 16B. In the information processing device 1b, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, question unit 113, and a control unit (not shown) are realized by the processing circuit 1001. In other words, when the information processing device 1b displays content on the display unit 201, it includes a processing circuit 1001 for controlling the system to ask questions to acquire the subject's reaction to the displayed content and to acquire the answers to those questions as reaction information. The processing circuit 1001 may be dedicated hardware as shown in Figure 16A, or it may be a processor 1004 that executes a program stored in memory as shown in Figure 16B.

[0163] The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the questioning unit 113, and a control unit (not shown). In other words, the information processing device 1b includes a memory 1005 for storing a program that, when executed by the processing circuit 1001, will result in the execution of steps ST10 to ST100 in Figure 7 described above. Furthermore, the program stored in memory 1005 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the questioning unit 113, and the control unit (not shown). Furthermore, the information processing device 1b includes information output devices 2, 2a, 2b, 2c, activity entity DB3, content server 4, and fan profile DB5, as well as an input interface device 1002 and an output interface device 1003 for wired or wireless communication.

[0164] If the information processing device is an information processing device 1c as shown in Figure 14, the hardware configuration of the information processing device 1c is also the configuration shown using Figures 16A and 16B. In the information processing device 1c, the functions of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, position optimization unit 114, and a control unit (not shown) are realized by the processing circuit 1001. In other words, the information processing device 1c includes a processing circuit 1001 for controlling the generation of position change information to change the positions of information output devices 2, 2a, 2b, and 2c based on the measured pedestrian flow, and for measuring pedestrian flow based on the captured image. The processing circuit 1001 may be dedicated hardware as shown in Figure 16A, or it may be a processor 1004 that executes a program stored in memory as shown in Figure 16B.

[0165] The processing circuit 1001 reads and executes a program stored in the memory 1005, thereby executing the functions of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the position optimization unit 114, and a control unit (not shown). In other words, the information processing device 1c includes a memory 1005 for storing a program that, when executed by the processing circuit 1001, will result in the execution of steps ST10 to ST105 in Figure 15 described above. Furthermore, the program stored in memory 1005 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the object recognition unit 102, the orientation estimation unit 103, the content selection unit 104, the content acquisition unit 105, the display control unit 106, the reaction information acquisition unit 107, the effect estimation unit 108, the model update unit 109, the profile update unit 111, the position optimization unit 114, and the control unit (not shown). Furthermore, the information processing device 1c includes information output devices 2, 2a, 2b, 2c, activity entity DB3, content server 4, and fan profile DB5, as well as an input interface device 1002 and an output interface device 1003 for wired or wireless communication.

[0166] In the above embodiment 1, the imaging unit 202 is assumed to be provided in the information output devices 2, 2a, 2b, and 2c, but this is merely an example. The imaging unit 202 may be provided outside the information output devices 2, 2a, 2b, and 2c. The imaging unit 202 only needs to be provided so as to be able to image an area that includes at least the front of the display unit 201.

[0167] Furthermore, in the above embodiment 1, the content selection unit 104 in the information processing devices 1, 1a, 1b, and 1c selects content using a reinforcement learning model. However, this is merely an example, and the content selection unit 104 may select content by other means. For example, the content selection unit 104 may select content according to pre-set conditions (hereinafter referred to as "selection conditions"). The selection conditions are set in advance by an administrator or the like. When the administrator or the like sets the selection conditions, they store information indicating the set selection conditions (hereinafter referred to as "selection condition information") in a storage unit not shown. The administrator or the like may update the selection condition information as appropriate. In this case, the information processing devices 1, 1a, 1b, and 1c will not include the effect estimation unit 108, the model update unit 109, and the model storage unit 110. Furthermore, the information processing devices 1, 1a, 1b, and 1c may also be configured without the reaction information acquisition unit 107 and the profile update unit 111.

[0168] Furthermore, in the above embodiment 1, the information processing devices 1, 1a, 1b, and 1c were assumed to be mounted on a server, for example, but this is merely one example. For example, the information processing devices 1, 1a, 1b, and 1c may be located in places other than the server, such as the information output devices 2, 2a, 2b, and 2c, or PCs located in a control room or the like. Also, for example, some of the image acquisition unit 101, object recognition unit 102, orientation estimation unit 103, content selection unit 104, content acquisition unit 105, display control unit 106, reaction information acquisition unit 107, effect estimation unit 108, model update unit 109, profile update unit 111, attribute determination unit 112, question unit 113, and position optimization unit 114 may be mounted on the server, while the others are located in places other than the server, and the system may be composed of the server and the locations other than the server.

[0169] As described above, according to Embodiment 1, the information processing devices 1, 1a, 1b, and 1c are configured to include: an image acquisition unit 101 that acquires captured images of a subject; an object recognition unit 102 that recognizes the subject's possessions or clothing based on the captured images acquired by the image acquisition unit 101 and detects existing trademarks or designs contained in the recognized possessions or clothing; an orientation estimation unit 103 that estimates an organization or person of interest or concern to the subject based on object recognition information indicating the trademark or design detected by the object recognition unit 102; a content selection unit 104 that selects content to be provided from content related to the organization or person stored in the content server 4 based on the orientation information indicating the organization or person estimated by the orientation estimation unit 103; a content acquisition unit 105 that acquires the content selected by the content selection unit 104 from the content server 4; and a display control unit 106 that displays the content acquired by the content acquisition unit 105 on the display unit 201. Therefore, the information processing devices 1, 1a, 1b, and 1c can provide information to the target person that is tailored to the organizations or individuals that the target person is interested in. The information processing devices 1, 1a, 1b, and 1c can provide the target person with content that is highly likely to attract their interest.

[0170] Furthermore, according to Embodiment 1, in the information processing devices 1, 1a, 1b, and 1c, the content selection unit 104 may be configured to select content related to the group or person estimated by the orientation estimation unit 103 based on orientation information if it determines that the target person is a specific target person who is eligible to receive content tailored to their interests, and to select content set as normal content if it determines that the target person is not a specific target person. Therefore, the information processing devices 1, 1a, 1b, and 1c can target individuals who are particularly interested in or concerned with a specific organization or person, and provide information that more effectively promotes that organization or person.

[0171] Furthermore, according to Embodiment 1, in the information processing devices 1, 1a, 1b, and 1c, the content selection unit 104 may be configured to include a response information acquisition unit 107 that selects content using a reinforcement learning model and acquires response information regarding the subject's response to the content displayed on the display unit 201, an effect estimation unit 108 that calculates a reward to be given to the reinforcement learning model based on the response information acquired by the response information acquisition unit 107 and reward conditions, and a model update unit 109 that updates the reinforcement learning model based on orientation information, information indicating the content displayed on the display unit 201 and reward information indicating the reward calculated by the effect estimation unit 108. Therefore, the information processing devices 1, 1a, 1b, and 1c can improve the accuracy of content selection when providing information to a target person that is tailored to the organizations or individuals that the target person is interested in.

[0172] Furthermore, according to Embodiment 1, in an information processing device 1b that includes the above-mentioned reaction information acquisition unit 107, effect estimation unit 108, and model update unit 109, when the display control unit 106 displays content on the display unit 201, the device may include a questioning unit 113 that asks questions to acquire the subject's reaction to the displayed content, and the reaction information acquisition unit 107 may acquire the answers to the questions asked by the questioning unit 113 as reaction information. Therefore, the information processing device 1b can further improve the accuracy of content selection when providing information to a target person that is tailored to the organizations or individuals that the target person is interested in.

[0173] Furthermore, according to Embodiment 1, the information processing device 1a may include an attribute determination unit 112 that determines the attributes of the subject, including the subject's gender, age, or whether or not a relationship with other subjects in the captured image is recognized, based on the captured image acquired by the image acquisition unit 101, and the content selection unit 104 may be configured to select content based on orientation information and attribute information indicating attributes. Therefore, the information processing device 1a can provide information to the target person in a more appropriate manner, taking into account the target person's attributes, that is tailored to the organizations or individuals that the target person is interested in or concerned with.

[0174] Furthermore, according to Embodiment 1, in the information processing devices 1, 1a, 1b, and 1c, the orientation estimation unit 103 may be configured to estimate an organization or person of interest to the subject if the object recognition unit 102 fails to detect a trademark or design, based on profile information including information about trademarks or designs previously detected for the subject, content previously displayed to the subject, or information about past activities of the subject related to an organization or person. Therefore, if the information processing devices 1, 1a, 1b, and 1c are unable to detect a trademark or design from the subject's possessions or clothing at present, they can estimate the group or person that is of interest or concern to the subject based on trademarks or designs previously detected from the subject's possessions or clothing, or from the subject's past activities.

[0175] Furthermore, according to Embodiment 1, the information processing device 1c may also include a position optimization unit 114 that measures pedestrian flow based on captured images and generates position change information for changing the position of the display unit 201 based on the measured pedestrian flow. Therefore, the information processing device 1c can improve the efficiency of information provision so that it can be used to its fullest potential in the most optimal location.

[0176] Furthermore, according to Embodiment 1, the information processing system 10 is configured to include an image acquisition unit 101 that acquires captured images of a subject, an object recognition unit 102 that recognizes the subject's possessions or clothing based on the captured images acquired by the image acquisition unit 101 and detects existing trademarks or designs contained in the recognized possessions or clothing, an orientation estimation unit 103 that estimates an organization or person that is the subject's interest or concern based on object recognition information indicating the trademark or design detected by the object recognition unit 102, a content selection unit 104 that selects content to be provided from content related to the organization or person stored in the content server 4 based on the orientation information indicating the organization or person estimated by the orientation estimation unit 103, a content acquisition unit 105 that acquires the content selected by the content selection unit 104 from the content server 4, a display control unit 106 that displays the content acquired by the content acquisition unit 105 on the display unit 201, and information output devices 2, 2a, 2b, 2c having the display unit 201. Therefore, the information processing system 10 can provide information to the target person that is tailored to the organizations or individuals that the target person is interested in or concerned about.

[0177] Furthermore, according to Embodiment 1, in the information processing system 10, the information output device 2b may be configured to have a character section 205 that represents an organization or a person. Therefore, the information processing system 10 can provide information in a more personal way.

[0178] Furthermore, according to Embodiment 1, in the information processing system 10, the information output devices 2a and 2b may be configured to have a microphone 203 and a speaker 204, and when the display control unit 106 displays content on the display unit 201, a question unit 113 outputs a question via voice from the speaker 204 to obtain the subject's response to the displayed content, and a response information acquisition unit 107 acquires the spoken voice in response to the question collected by the microphone 203 as response information. Therefore, the information processing system 10 can provide information to the target audience in a way that offers a more natural and intuitive user experience.

[0179] Furthermore, according to Embodiment 1, in an information processing system 10 having the above-mentioned questioning unit 113 and response information acquisition unit 107, the information output device 2b may have a character unit 205 representing a group or person, and the microphone 203 and speaker 204 may be provided on the character unit 205. Therefore, the information processing system 10 can provide information to the target audience in a more natural and intuitive user experience, and can also provide such information in a more intimate way.

[0180] Furthermore, according to Embodiment 1, in the information processing system 10, the information output device 2c may have a moving mechanism 206 and a position optimization unit 114 that measures pedestrian flow based on captured images and generates position change information for changing the position of the information output device 2c based on the measured pedestrian flow, and the moving mechanism 206 may be configured to drive according to the position change information generated by the position optimization unit 114. Therefore, the information processing system 10 can improve the efficiency of information provision so that it can be most effective in the optimal location.

[0181] The various aspects of this disclosure are summarized below as an appendix.

[0182] (Note 1) An image acquisition unit that acquires captured images of the subject, Based on the image captured by the image acquisition unit, the object recognition unit recognizes the subject's possessions or clothing and detects any existing marks or designs contained in the recognized possessions or clothing. Based on the object recognition information indicating the mark or design detected by the object recognition unit, the orientation estimation unit estimates the group or person that is the object of interest or concern of the subject, A content selection unit selects the content to be provided from among the content related to the organization or person stored in the content server, based on the orientation information indicating the organization or person estimated by the orientation estimation unit. A content acquisition unit that acquires the content selected by the content selection unit from the content server, Display control unit that displays the content acquired by the content acquisition unit on the display unit. An information processing device equipped with the following. (Note 2) The aforementioned content selection unit is Based on the estimation results of the orientation estimation unit for the group or person, if it is determined that the target person is a specific target person to whom the content corresponding to the target person's interests or concerns should be provided, the unit selects the content related to the group or person estimated by the orientation estimation unit based on the orientation information. If it is determined that the target person is not a specific target person, the unit selects the content that is normally set as the content. The information processing apparatus described in Appendix 1, characterized by the features described herein. (Note 3) The content selection unit selects the content using a reinforcement learning model. A reaction information acquisition unit that acquires reaction information regarding the subject's reaction to the content displayed on the display unit, An effect estimation unit calculates the reward to be given to the reinforcement learning model based on the reaction information acquired by the reaction information acquisition unit and the reward conditions, Model update unit updates the reinforcement learning model based on the orientation information, the information indicating the content displayed on the display unit, and the reward information indicating the reward calculated by the effect estimation unit. An information processing apparatus according to Appendix 1 or Appendix 2, comprising the above. (Note 4) When the display control unit causes the display unit to display the content, it includes a questioning unit that asks questions to obtain the subject's response to the displayed content. The response information acquisition unit acquires the response to the question asked by the questioning unit as the response information. The information processing apparatus described in Appendix 3, characterized by the features described herein. (Note 5) The image acquisition unit has an attribute determination unit that determines the attributes of the subject, including the subject's gender, age, or whether or not a relationship with other subjects in the captured image is recognized, based on the captured image acquired by the image acquisition unit. The content selection unit selects the content based on the orientation information and the attribute information indicating the attributes. An information processing apparatus characterized by any one of the appendices 1 to 4. (Note 6) The aforementioned directivity estimation unit, If the object recognition unit fails to detect the trademark or design, it estimates the organization or person that is the subject of the subject's interest or concern from profile information including information about trademarks or designs previously detected for the subject, content previously displayed to the subject, or information about past activities of the subject related to the organization or person. An information processing device characterized by any one of the appendices 1 to 5. (Note 7) A position optimization unit measures pedestrian flow based on the captured image and generates position change information for changing the position of the display unit based on the measured pedestrian flow. An information processing device described in any one of the appendices 1 to 5, comprising the features described therein. (Note 8) Computers, An image acquisition unit that acquires captured images of the subject, Based on the image captured by the image acquisition unit, the object recognition unit recognizes the subject's possessions or clothing and detects any existing marks or designs contained in the recognized possessions or clothing. An orientation estimation unit that estimates a group or a person that is the target of the subject's interest or concern based on the object recognition information indicating the trademark or the design detected by the object recognition unit, A content selection unit that selects the content to be provided from the contents related to the group or the person stored in the content server based on the orientation information indicating the group or the person estimated by the orientation estimation unit, A content acquisition unit that acquires the content selected by the content selection unit from the content server, A display control unit that causes the display unit to display the content acquired by the content acquisition unit An information processing program for causing it to function. (Appendix 9) A step in which an image acquisition unit acquires a captured image of a subject, A step in which an object recognition unit recognizes an object or an article worn by the subject based on the captured image acquired by the image acquisition unit, and detects a trademark or a design existing in the recognized object or article worn, A step in which an orientation estimation unit estimates a group or a person that is the target of the subject's interest or concern based on the object recognition information indicating the trademark or the design detected by the object recognition unit, A step in which a content selection unit selects the content to be provided from the contents related to the group or the person stored in the content server based on the orientation information indicating the group or the person estimated by the orientation estimation unit, A step in which a content acquisition unit acquires the content selected by the content selection unit from the content server, A step in which a display control unit causes the display unit to display the content acquired by the content acquisition unit An information processing method comprising the above. (Appendix 10) An image acquisition unit that acquires a captured image of a subject, Based on the captured image acquired by the image acquisition unit, recognize the belongings or clothing of the target person, and detect the existing trademarks or designs included in the recognized belongings or clothing, which is an object recognition unit; Based on the object recognition information indicating the trademark or the design detected by the object recognition unit, estimate the group or person that is the object of the interest or concern of the target person, which is an orientation estimation unit; Based on the orientation information indicating the group or the person estimated by the orientation estimation unit, select the content to be provided from the contents related to the group or the person stored in the content server, which is a content selection unit; A content acquisition unit that acquires the content selected by the content selection unit from the content server; A display control unit that causes the display unit to display the content acquired by the content acquisition unit; An information output device having the display unit An information processing system comprising the above. (Appendix 11) The display unit is a sign The information processing system according to Appendix 10, characterized in that. (Appendix 12) The information output device has an imaging unit for imaging the target person, The image acquisition unit acquires the captured image of the target person captured by the imaging unit The information processing system according to Appendix 10 or Appendix 11, characterized in that. (Appendix 13) The information output device has a character unit representing the group or the person The information processing system according to any one of Appendices 10 to 12, characterized in that. (Appendix 14) The information output device has a microphone and a speaker, When the display control unit causes the display unit to display the content, a question unit that outputs, by voice from the speaker, a question for acquiring the reaction of the target person to the displayed content; The microphone has a response information acquisition unit that acquires the spoken audio in response to the question as response information. An information processing system characterized by any one of the appendices 10 to 13. (Note 15) The information output device has a character section that represents the organization or the person, The microphone and speaker are provided in the character section. The information processing apparatus described in Appendix 14, characterized by the features described herein. (Note 16) The information output device has a moving mechanism, The system includes a position optimization unit that measures pedestrian flow based on the captured image and generates position change information for changing the position of the information output device based on the measured pedestrian flow, The moving mechanism is driven according to the position change information generated by the position optimization unit. An information processing system comprising the features described in any one of the appendices 10 to 15. (Note 17) The aforementioned content selection unit is Based on the estimation results of the orientation estimation unit for the group or person, if it is determined that the target person is a specific target person to whom the content corresponding to the target person's interests or concerns should be provided, the unit selects the content related to the group or person estimated by the orientation estimation unit based on the orientation information. If it is determined that the target person is not a specific target person, the unit selects the content that is normally set as the content. An information processing system characterized by any one of the appendices 10 to 16. (Note 18) The content selection unit selects the content using a reinforcement learning model. A reaction information acquisition unit that acquires reaction information regarding the subject's reaction to the content displayed on the display unit, An effect estimation unit calculates the reward to be given to the reinforcement learning model based on the reaction information acquired by the reaction information acquisition unit and the reward conditions, Model update unit updates the reinforcement learning model based on the orientation information, the information indicating the content displayed on the display unit, and the reward information indicating the reward calculated by the effect estimation unit. An information processing system comprising the features described in any one of the appendices 10 to 16. (Note 19) When the display control unit causes the display unit to display the content, it includes a questioning unit that asks questions to obtain the subject's response to the displayed content. The response information acquisition unit acquires the response to the question asked by the questioning unit as the response information. The information processing system described in Appendix 18, characterized by the features described herein. (Note 20) The image acquisition unit has an attribute determination unit that determines the attributes of the subject, including the subject's gender, age, or whether or not a relationship with other subjects in the captured image is recognized, based on the captured image acquired by the image acquisition unit. The content selection unit selects the content based on the orientation information and the attribute information indicating the attributes. An information processing system characterized by any one of the appendices 10 to 19. (Note 21) The aforementioned directivity estimation unit, If the object recognition unit fails to detect the trademark or design, it estimates the organization or person that is the subject of the subject's interest or concern from profile information including information about trademarks or designs previously detected for the subject, content previously displayed to the subject, or information about past activities of the subject related to the organization or person. An information processing system characterized by any one of the appendices 10 to 20. (Note 22) A position optimization unit measures pedestrian flow based on the captured image and generates position change information for changing the position of the display unit based on the measured pedestrian flow. An information processing system described in any one of the appendices 10 to 21, comprising the features of: [Explanation of Symbols]

[0183] 1,1a,1b,1c Information processing device, 101 Image acquisition unit, 102 Object recognition unit, 103 Directionality estimation unit, 104 Content selection unit, 105 Content acquisition unit, 106 Display control unit, 107 Reaction information acquisition unit, 108 Effect estimation unit, 109 Model update unit, 110 Model storage unit, 111 Profile update unit, 112 Attribute determination unit, 113 Question unit, 114 Position optimization unit, 2,2a,2b,2c Information output device, 201 Display unit, 202 Imaging unit, 203 Microphone, 204 Speaker, 205 Character unit, 206 Movement mechanism, 3 Activity entity DB, 4 Content server, 5 Fan profile DB, 10 Information processing system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. An image acquisition unit that acquires captured images of the subject, Based on the image captured by the image acquisition unit, the object recognition unit recognizes the subject's possessions or clothing and detects any existing marks or designs contained in the recognized possessions or clothing. Based on the object recognition information indicating the mark or design detected by the object recognition unit, the orientation estimation unit estimates the group or person that is the object of interest or concern of the subject, A content selection unit selects the content to be provided from among the content related to the organization or person stored in the content server, based on the orientation information indicating the organization or person estimated by the orientation estimation unit. A content acquisition unit that acquires the content selected by the content selection unit from the content server, Display control unit that displays the content acquired by the content acquisition unit on the display unit. An information processing device equipped with the following.

2. The aforementioned content selection unit is Based on the estimation results of the orientation estimation unit for the group or person, if it is determined that the target person is a specific target person to whom the content corresponding to the target person's interests or concerns should be provided, the unit selects the content related to the group or person estimated by the orientation estimation unit based on the orientation information. If it is determined that the target person is not a specific target person, the unit selects the content that is normally set as the content. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

3. The content selection unit selects the content using a reinforcement learning model. A reaction information acquisition unit that acquires reaction information regarding the subject's reaction to the content displayed on the display unit, An effect estimation unit calculates the reward to be given to the reinforcement learning model based on the reaction information acquired by the reaction information acquisition unit and the reward conditions, Model update unit updates the reinforcement learning model based on the orientation information, the information indicating the content displayed on the display unit, and the reward information indicating the reward calculated by the effect estimation unit. The information processing apparatus according to claim 1, comprising the above.

4. When the display control unit causes the display unit to display the content, it includes a questioning unit that asks questions to obtain the subject's response to the displayed content. The response information acquisition unit acquires the response to the question asked by the questioning unit as the response information. The information processing apparatus according to claim 3, characterized in that it is a product of the same method.

5. The image acquisition unit has an attribute determination unit that determines the attributes of the subject, including the subject's gender, age, or whether or not a relationship with other subjects in the captured image is recognized, based on the captured image acquired by the image acquisition unit. The content selection unit selects the content based on the orientation information and the attribute information indicating the attributes. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

6. The aforementioned directivity estimation unit, If the object recognition unit fails to detect the trademark or design, it estimates the organization or person that is the subject of the subject's interest or concern from profile information including information about trademarks or designs previously detected for the subject, content previously displayed to the subject, or information about past activities of the subject related to the organization or person. The information processing apparatus according to claim 1, characterized in that it is a product of the present invention.

7. A position optimization unit measures pedestrian flow based on the captured image and generates position change information for changing the position of the display unit based on the measured pedestrian flow. The information processing apparatus according to claim 1, comprising:

8. Computers, An image acquisition unit that acquires captured images of the subject, Based on the image captured by the image acquisition unit, the object recognition unit recognizes the subject's possessions or clothing and detects any existing marks or designs contained in the recognized possessions or clothing. Based on the object recognition information indicating the mark or design detected by the object recognition unit, the orientation estimation unit estimates the group or person that is the object of interest or concern of the subject, A content selection unit selects the content to be provided from among the content related to the organization or person stored in the content server, based on the orientation information indicating the organization or person estimated by the orientation estimation unit. A content acquisition unit that acquires the content selected by the content selection unit from the content server, Display control unit that displays the content acquired by the content acquisition unit on the display unit. An information processing program designed to function as such.

9. The image acquisition unit acquires the captured image of the subject, The object recognition unit recognizes the subject's possessions or clothing based on the image acquired by the image acquisition unit, and detects any existing marks or designs contained in the recognized possessions or clothing. The orientation estimation unit estimates an organization or person that is the subject of interest or concern of the subject, based on the object recognition information indicating the mark or design detected by the object recognition unit. The content selection unit selects the content to be provided from among the content related to the organization or person stored in the content server, based on the orientation information indicating the organization or person estimated by the orientation estimation unit. The content acquisition unit acquires the content selected by the content selection unit from the content server, The display control unit causes the content acquired by the content acquisition unit to be displayed on the display unit. An information processing method equipped with the following.

10. An image acquisition unit that acquires captured images of the subject, Based on the image captured by the image acquisition unit, the object recognition unit recognizes the subject's possessions or clothing and detects any existing marks or designs contained in the recognized possessions or clothing. Based on the object recognition information indicating the mark or design detected by the object recognition unit, the orientation estimation unit estimates the group or person that is the object of interest or concern of the subject, A content selection unit selects the content to be provided from among the content related to the organization or person stored in the content server, based on the orientation information indicating the organization or person estimated by the orientation estimation unit. A content acquisition unit that acquires the content selected by the content selection unit from the content server, A display control unit that causes the content acquired by the content acquisition unit to be displayed on the display unit, Information output device having the display unit An information processing system equipped with the following.

11. The aforementioned display unit is a digital signage. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

12. The information output device has an imaging unit for imaging the subject, The image acquisition unit acquires the captured image of the subject captured by the imaging unit. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

13. The information output device has a character section that represents the organization or the person. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

14. The information output device has a microphone and a speaker. When the display control unit causes the display unit to display the content, the question unit outputs a question via the speaker in voice to obtain the subject's response to the displayed content. The microphone has a response information acquisition unit that acquires the spoken audio in response to the question as response information. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

15. The information output device has a character section that represents the organization or the person, The microphone and speaker are provided in the character section. The information processing system according to claim 14, characterized in that it is the same as described in claim 14.

16. The information output device has a moving mechanism, The system includes a position optimization unit that measures pedestrian flow based on the captured image and generates position change information for changing the position of the information output device based on the measured pedestrian flow, The moving mechanism is driven according to the position change information generated by the position optimization unit. The information processing system according to claim 10, comprising the above.

17. The aforementioned content selection unit is Based on the estimation results of the orientation estimation unit for the group or person, if it is determined that the target person is a specific target person to whom the content corresponding to the target person's interests or concerns should be provided, the unit selects the content related to the group or person estimated by the orientation estimation unit based on the orientation information. If it is determined that the target person is not a specific target person, the unit selects the content that is normally set as the content. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

18. The content selection unit selects the content using a reinforcement learning model. A reaction information acquisition unit that acquires reaction information regarding the subject's reaction to the content displayed on the display unit, An effect estimation unit calculates the reward to be given to the reinforcement learning model based on the reaction information acquired by the reaction information acquisition unit and the reward conditions, Model update unit updates the reinforcement learning model based on the orientation information, the information indicating the content displayed on the display unit, and the reward information indicating the reward calculated by the effect estimation unit. The information processing system according to claim 10, comprising the above.

19. When the display control unit causes the display unit to display the content, it includes a questioning unit that asks questions to obtain the subject's response to the displayed content. The response information acquisition unit acquires the response to the question asked by the questioning unit as the response information. The information processing system according to claim 18, characterized in that it is the same as described above.

20. The image acquisition unit has an attribute determination unit that determines the attributes of the subject, including the subject's gender, age, or whether or not a relationship with other subjects in the captured image is recognized, based on the captured image acquired by the image acquisition unit. The content selection unit selects the content based on the orientation information and the attribute information indicating the attributes. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

21. The aforementioned directivity estimation unit, If the object recognition unit fails to detect the trademark or design, it estimates the organization or person that is the subject of the subject's interest or concern from profile information including information about trademarks or designs previously detected for the subject, content previously displayed to the subject, or information about past activities of the subject related to the organization or person. The information processing system according to claim 10, characterized in that it is the same as described in claim 10.

22. A position optimization unit measures pedestrian flow based on the captured image and generates position change information for changing the position of the display unit based on the measured pedestrian flow. The information processing system according to claim 10, comprising:

Images